Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14552

Über-jira: WASB client phase II: performance and testing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.1
    • 2.9.0, 3.0.0
    • fs/azure
    • None

    Description

      Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9.

      I think the focus is on performance, but looking at the tests there are some things we need there first: parallel test execution, move over to IT tests, other tuning.

      All patches must come with declarations of which azure endpoint they were tested against.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HADOOP-14547 [WASB] the configured retry policy is not used for all storage operations.

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14518 Customize User-Agent header sent in HTTP/HTTPS requests by WASB.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14461 Azure: handle failure gracefully in case of missing account access key

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14478 Optimize NativeAzureFsInputStream for positional reads

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14581 Restrict setOwner to list of user when security is enabled in wasb

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14598 Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14443 Azure: Support retry and client side failover for authorization, SASKey and delegation token generation

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14520 WASB: Block compaction for Azure Block Blobs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14535 wasb: implement high-performance random access and seek of block blobs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14536 Update azure-storage sdk to version 5.3.0

          • Major - Major loss of function.
          • Resolved
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15132 Über-jira: WASB client phase III: roll-up for Hadoop 3.2

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14478 Optimize NativeAzureFsInputStream for positional reads

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14680 Azure: IndexOutOfBoundsException in BlockBlobInputStream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15042 Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          1.
          		 Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Sub-task Resolved Georgi Chalakov
          2.
          		 Upgrade azure-storage sdk version >5.4.0 Sub-task Resolved Rajesh Balamohan
          3.
          		 Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients Sub-task Closed Unassigned
          4.
          		 Remove service loader config file for wasb fs Sub-task Resolved Esfandiar Manii
          5.
          		 Add (parallelized) integration tests to hadoop-azure Sub-task Resolved Steve Loughran
          6.
          		 TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest Sub-task Resolved Unassigned
          7.
          		 document how to run wasb tests in azure docs site/testing.md Sub-task Resolved Unassigned
          8.
          		 [WASB] the configured retry policy is not used for all storage operations. Sub-task Resolved Thomas Marqardt
          9.
          		 regression: Azure tests which capture logs failing with move to SLF4J Sub-task Resolved Steve Loughran
          10.
          		 wasb throws an exception if you try to create a file and there's no parent directory Sub-task Closed Esfandiar Manii
          11.
          		 Azure: Concurrent I/O does not work when secure.mode is enabled Sub-task Resolved Thomas Marqardt
          12.
          		 Azure: Reuse ObjectMapper Sub-task Resolved Mingliang Liu
          13.
          		 NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code Sub-task Resolved Steve Loughran
          14.
          		 error message in AzureNativeFileSystemStore.connectUsingAnonymousCredentials has too many spaces Sub-task Resolved Mingliang Liu
          15.
          		 wasb: add support for caching Authorization and SASKeys Sub-task Resolved Sivaguru Sankaridurg
          16.
          		 wasb: improve throughput by 34% when account limit exceeded Sub-task Resolved Thomas Marqardt
          17.
          		 TestWasbRemoteCallHelper failing Sub-task Resolved Esfandiar Manii
          18.
          		 Honoring sticky bit during Deletion when authorization is enabled in WASB Sub-task Resolved Varada Hemeswari
          19.
          		 WASB: delete recursive should not fail if a file is deleted Sub-task Resolved Thomas Marqardt
          20.
          		 PageBlobOutputStream to declare hflush/hsync StreamCapabilities for Sub-task Resolved Unassigned
          21.
          		 Add support for using container saskeys for all accesses Sub-task Resolved Sivaguru Sankaridurg
          22.
          		 Azure wasb: getFileStatus not making any auth checks Sub-task Resolved Sivaguru Sankaridurg
          23.
          		 Restrict Access to setPermission operation when authorization is enabled in WASB Sub-task Resolved Kannapiran Srinivasan
          24.
          		 Azure: POSIX permissions are taking effect in access() method even when authorization is enabled Sub-task Resolved Santhosh G Nayak
          25.
          		 Sticky bit implementation for rename() operation in Azure WASB Sub-task Resolved Varada Hemeswari
          26.
          		 ITestNativeAzureFileSystemLive.testLeaseAsDistributedLock assert failure in parallel run Sub-task Resolved Unassigned

          Activity

            People

              tmarquardt Thomas Marqardt
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: