Uploading large files

In the DEXTUpload family of products, uploading files with a size of 2 GB or more is referred to as a "Large file upload". DEXTUploadX5 provides the ability to handle the large file upload in conjunction with our server component products, such as DEXTUploadNJ (based on the Java environment). (Large file upload is only possible in conjunction with our components and is not supported by DEXTUploadX5 alone.)

Large file upload is handled differently than a normal file upload. While the normal file upload can send all 10 files in a single request, the large file upload can send at least 20 and sometimes thousands of requests to the server, depending on the situation. Since DEXTUploadX5's large file upload feature natively supports concatenation, the product communicates with the server component in advance, and since it sends the file in pieces, the number of requests increases with the size of the pieces (chunks). For example, if you want to upload 10 files of 2.5 GB in size and transfer them in chunks of about 10 MB each, you can expect to make about 2500 requests. However, since server components like DEXTUploadNJ take care of combining the chunks into a single file, the developer only has to deal with storing (transferring) the file to the destination about 10 times in total.

How to set up

To implement the large file upload service, both the client and server components must be configured for the large file upload.

The following is an example of DEXTUploadX5 configuration.

// Set up in the onDX5Created event function.
function onDX5Created(id) {
    var dx = dx5.get(id);

    // Set the upload path.
    dx.setUploadURL("http://domain/path/extension-upload.do");

    // Set the upload mode to bulk.
    dx.setUploadMode("EXTS");

    // Set the block size, in bytes, to split the file into.
    dx.setUploadBlockSize(10 * 1024 * 1024);
}

Server-side setup using the DEXTUploadNJ server component product

DEXTUploadNJ is a server-only component that handles file uploads on a Java JSP/servlet basis. DEXTUploadNJ includes features for server-side processing of the large file upload and works with our DEXTUploadX5 client suite.

The ExtensionFileUploadFilter class is an implementation of the javax.servlet.Filter interface. Because it is a Filter class, it is automatically loaded by a Java servlet container (like a Tomcat server) and handles the large file upload without any code-level creation or invocation.

Here's how to set up the ExtensionFileUploadFilter filter in DD(web.xml)

<filter>
  <filter-name>extensionFilter</filter-name>
  <filter-class>devpia.dextuploadnj.support.common.ExtensionFileUploadFilter</filter-class>
  <!-- omit setting parameters -->
</filter>
<filter-mapping>
  <filter-name>extensionFilter</filter-name>
  <!-- 
  A servlet or URL mapping to finally handle the large file upload at the developer code level
  <servlet-name>Servlet name</servlet-name>
  <url-pattern>Mapping URL</url-pattern>
  -->
</filter-mapping>

Server-side processing (JSP/Servlet) using DEXTUploadNJ (before 2.5.0) server component product

The file information whose upload is processed through the ExtensionFileUploadFilter filter can be obtained by using the HttpServletRequest#getAttribute method in the servlet or JSP that the developer needs to write.

// Use DEXTUPLOADNJ_EXTENSION_FILE_UPLOAD_RESULT as a key to get a MultipartCollection object that contains the filter's processing results.
MultipartCollection multiparts = (MultipartCollection)request.getAttribute(Definition.DEXTUPLOADNJ_EXTENSION_FILE_UPLOAD_RESULT);
try {
    FileItem item = multiparts.getFileItem(0);
    
    if (item.isEmpty()) {
        // If the file is empty, you should generally throw an exception.
    }
    
    // Save (copy or move) the temporary file to the actual storage location.
    // If no directory path is given as an argument, the path set by the value of the defaultRepository parameter of the ExtensionFileUploadFilter will be targeted.
    String path = item.save();
        
    response.setCharacterEncoding("UTF-8");
    response.setContentType("text/plain");
    // You can write any value to the response object, but it's common to write a key or a file name or path that represents a unique value for the file.
    response.getWriter().write(path);
} finally {
    if (multiparts != null) {
        // Remove the resource. Any temporary files that have not been deleted will be deleted.
        multiparts.deleteTempFiles();
        multiparts.clear();
    }
}

Server-side processing (JSP/Servlet) using the DEXTUploadNJ (2.5.0 and later) server component product

Starting with version 2.5.0, you can reduce the complicated preliminary steps and use the FileUpload class to get the file item.

FileUpload dextnj = null;
try {
    dextnj = new FileUpload(request);
    dextnj.prepare();

    FileItem item = dextnj.getFileItem();
    if (item.isEmpty()) {
        // If the file is empty, you should normally raise an exception.
    }
  
    String path = item.save();
  
    response.setCharacterEncoding("UTF-8");
    response.setContentType("text/plain");
    response.getWriter().write(path);
} finally {
    if (dextnj != null) dextnj.close();
}

Server-side processing using the DEXTUploadNJ (before 2.6.0) server component product (Spring Web Framework)

In the Spring environment, not only do we need to set the ExtensionFileUploadFilter in DD(web.xml), but we also need to set the MultipartResolver bean in the XML file responsible for setting the DispatcherServlet object.

# web.xml
<servlet>
  <servlet-name>defaultDispatcher</servlet-name>
  <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  <init-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/dispatcher-servlet.xml</param-value>
  </init-param>
</servlet>

Open the dispatcher-servlet.xml file to declare the DEXTUploadNJMultipartResolver bean.

# dispatcher-servlet.xml
<bean id="multipartResolver" class="devpia.dextuploadnj.support.spring.DEXTUploadNJMultipartResolver"/>

If you have CommonMultipartResolver or StandardServletMultipartResolver (provided by Spring) set up, you should delete them. These beans perform the same task as the DEXTUploadNJMultipartResolver bean, so setting them at the same time may cause an error.

The temporary file information processed through the ExtensionFileUploadFilter filter can be received as a MultipartFile object in the method mapped to the Controller, where the final saving operation can be performed.

@RequestMapping(value = "/extension-upload.do", method = RequestMethod.POST)
public void upload(
    // If this is a DEXTUploadX5 product, it will be passed as "DEXTUploadX5_FileData".
    @RequestParam(value = "DEXTUploadX5_FileData") MultipartFile file, 
    HttpServletResponse response) throws IOException {
    
    // Cast to the FileItem interface.
    FileItem item = (FileItem)file;	
    
    if (item.isEmpty()) {
        // If the file is empty, an exception should normally be thrown.
        // This is because a client like DEXTUploadX5 // will not send a request to the server with no file.
    }  
    
    String path = item.save();
    
    response.setCharacterEncoding("UTF-8");
    response.setContentType("text/plain");
    response.getWriter().write(path);
}

Server-side processing with the DEXTUploadNJ (2.6.0 and later) server component product (Spring Web Framework)

Starting with version 2.6.0, DEXTUploadNJ provides a dedicated filter, DEXTUploadNJSpringExtensionUploadFilter, for use in Spring environments. You can use this filter to omit the DEXTUploadNJMultipartResolver setting.

# web.xml
<filter>
  <filter-name>extensionUploadFilter</filter-name>
  <filter-class>devpia.dextuploadnj.support.spring.DEXTUploadNJSpringExtensionUploadFilter</filter-class>
  <!-- omit setting parameters -->
</filter>

# dispatcher-servlet.xml
<!-- You can omit the DEXTUploadNJMultipartResolver configuration. -->

Processing the response data received from the server

Once the server-side processing is complete, DEXTUploadX5 returns the results as response data. This response data can be obtained using the component's getResponses function. On the server, the code is called as many times as there are files, but the onDX5UploadCompleted event function is called only once because the upload completion event only occurs once.

// The onDX5UploadCompleted function is an event function that is called when the upload is complete (server-side file upload processing is complete).
function onDX5UploadCompleted(id) {
    var responses = dx5.get(id).getResponses();
    for (var i = 0, len = responses.length; i < len; i++) {
        console.log(responses[i]);
    }
}