March Update on Development Roadmap for 2017

Microsoft has released Visual Studio 2017, which is the development environment used for Gepsio development. Gepsio’s development roadmap for 2017, published in December 2016 and outlined here, noted that Gepsio would be moving to .NET Standard 2.0 once Visual Studio 2017 was released, as .NET Standard 2.0 would allow Gepsio to support a variety of environments, including the full .NET Framework, .NET Core, Xamarin.iOS, Xamarin.Android, and Linux.

The original roadmap was written on the assumption that .NET Standard 2.0 would be shipping “out of the box” with Visual Studio 2017. According to Microsoft’s .NET Core Roadmap, .NET Standard 2.0 and .NET Core 2.0 will not be released until Q3 2017, with public preview builds arriving in Q2 2017. Therefore, the project will remain in Codeplex and Visual Studio 2015 until the public previews of .NET Standard 2.0 are available.

To recap, once .NET Standard 2.0 becomes available, the project will move forward as follows:

  • the solution will migrate from Visual Studio 2015 to Visual Studio 2017
  • the Gepsio assembly will migrate from a .NET Framework target to a .NET Standard 2.0 target
  • the Gepsio unit tests will migrate from a .NET Framework target to a .NET Core 2.0 target
  • the repository will migrate from Codeplex over to Github
  • the repository access will migrate from TFS to git

For now, the work will remain at Codeplex and built by Visual Studio 2015. There have been several improvements made lately, with changeset history visible here, and it’s high time for a new release. Perhaps a new build can be pushed out to NuGet in April.

Development Roadmap for 2017

Visual Studio 2017 and .NET Standard promise to bring exciting changes to .NET-based development efforts, and Gepsio will be taking full advantage of these changes. Gepsio’s cross-platform vision, as outlined here, is to make use of .NET Standard to bring Gepsio’s functionality to the full .NET Framework, .NET Core, the Universal Windows Platform, Mono, and Xamarin for iOS and Android.

The Wait for .NET Standard 2.0

The big waiting game for moving on the cross-platform story is with .NET Standard 2.0. As discussed previously, .NET Standard 2.0 will contain support for the classes in the XML Schema namespace not available in earlier versions of .NET Standard. Since Gepsio heavily leverages XML Schema, .NET Standard support for XML Schema is a prerequisite for any migration of the code base over to .NET Standard. The good news is that the .NET Portability Analyzer has been run against the current Gepsio binary and its API usage was found to be 100% compatible with .NET Standard 2.0. The wait for the 2.0 release is necessary, because the code base is only 73.31% compatible with .NET Standard 1.4 and 73.68% compatible with .NET Standard 1.5 and 1.6 due to Gepsio’s use of classes in the XML Schema namespace. It is currently assumed that .NET Standard 2.0 will be released in the same time frame as Visual Studio 2017, which, according to current documentation, is currently slated for some time in the spring of 2017.

From Codeplex to GitHub

Once .NET Standard 2.0 becomes available, and the code base is migrated to the new platform, the project will find a new home at GitHub. There have been a few requests to move the project over to GitHub to allow Git clients to repo the solution, and the new .NET Standard reconfiguration seems like a good time to make the change. The historical project will remain at Codeplex, but, once the .NET Standard migration is complete, new code will be available at GitHub.
As a part of this change over to Git, the solution folders will be simplified to match de facto conventions used by other GitHub projects. There will be a few folders at the root of the solution:
  • src, where the source will be stored
  • test, where the unit tests will be stored
  • docs, for documentation and possibly a GitHub Pages site for the project
  • nuget, for NuGet specification files and other related information

Simplified Solution Structure

A previous post discussed a restructuring of the solution which would enable Gepsio to provide several different versions of the .NET Framework. The restructuring, based on a Visual Studio 2015 project type called Shared Projects, allowed one code base to be used in different referencing projects, each targeting a different version of .NET. With the move to .NET Standard 2.0, only one version of the .NET Framework — version 4.6.1 — will be supported, so all of the other various .NET Framework target folders will be removed.
The overall concept of a shared project folder containing the source, as well as other projects for various targets, will remain. It is expected that the solution will have projects for various platform-specific compilation targets, such as .NET Core, Mono, and Xamarin, but they should all be able to use the code from the main Shared Project containing the Gepsio source.

UWP

A previous post outlined the original thoughts behind Gepsio’s support for the Universal Windows Platform. At the time, it was envisioned that the solution would contain a separate implementation of the various XML interfaces usable by the UWP compilation target. Work was well underway when .NET Standard was announced. However, .NET Standard promises to support UWP with a standard implementation, so UWP-specific interface implementations for XML support are unneeded. Therefore, the UWP project will be removed and replaced with a UWP compilation project that supports .NET Standard.

Increased XBRL Conformance

At the end of the day, Gepsio’s raison d’être is to provide support for the XBRL specification. To be honest, this work has been sidelined in favor of building up cross-platform support. Once .NET Standard 2.0 is in place, and Gepsio’s code base is set up to offer that cross-platform support, work can continue on Gepsio’s support for the XBEL specification. This vision includes support for the various XBRL specifications, including Dimensions and Formulas.

Moving Gepsio Platform Support Forward with .NET Standard

Gepsio currently supports several different versions of the .NET Framework, but it has long been a goal to support the code on a variety of platforms, including the Universal Windows Platform (UWP) and Xamarin. Microsoft is making strides in the standardization of the .NET platform that should make this goal attainable while allowing the Gepsio codebase to move forward with relative ease into these new platforms.
Microsoft recently introduced .NET Core, and, with it, an API set called .NET Standard. .NET Standard is a set of APIs that all .NET platforms have to implement. As of this writing, the current version of .NET Standard is version 1.6, which supports the following platforms:
  • .NET Core 1.0
  • .NET Framework 4.6.2
  • Mono 4.6
  • Xamarin.iOS 10.0
  • Xamarin.Android 7.0
  • Universal Windows Platform 10.0
Some investigation has been performed on the Gepsio code base with the goal of understanding the impact of moving the Gepsio code base to .NET Standard 1.6. The good news is that the vast majority of the code base moved over without much of an issue, giving hope that the migration of the Gepsio code base away from its sole focus on the .NET Framework and into a broader view of the current platform landscape is achievable.

There is, however, one blocking issue which prevents Gepsio from adopting .NET Standard 1.6 today. Although .NET Standard 1.6 supports the System.Xml namespace, it does not support the System.Xml.Schema namespace. Since XBRL heavily leverages the XML Schema specification, and since Gepsio leverages data from XBRL schemas in its implementation, Gepsio must implement XML Schema support in its codebase in the absence of support from .NET Standard. With System.Xml.Schema unavailable, Gepsio will not be able to leverage the classes in the namespace using .NET Standard 1.6.

There are, at this point, two options for Gepsio:

  • Provide an independent implementation of XML Schema. Gepsio could, if necessary, provide a “homegrown” implementation of XML Schema support. Since an XML Schema document is, at its core, an XML document, Gepsio could leverage the classes available in System.Xml to read in a schema document and parse it using the requirements in the XML Schema specification. This work would become non-trivial, as the implementation would need to support the specification’s type system and notions of simple and complex types, as well as exposing all of that information to the pieces of Gepsio which require schema information. In this plan, Gepsio would be required to implement the XML Schema implementation “from scratch”.
  • Wait for XML Schema support in a future version of .NET Standard. Gepsio could also do nothing and wait for the .NET Standard to support the System.Xml.Schema namespace and then move over to that version of the Standard at that time.
The second option seems to be a more practical approach, since it would not require that an entire XML Schema implementation be written from the ground up. The question is: how long will it take for .NET Standard to provide that implementation?
As it turns out, not too long.
The next version of .NET Standard, labeled 2.0, is slated to support the XML Schema namespace. As of this writing, 924 APIs from the System.Xml.Schema namespace will be supported. Work on .NET Standard is already underway and is slated to support the following platforms:
  • .NET Core vNext
  • .NET Framework 4.6.1
  • Mono vNext
  • Xamarin.iOS vNext
  • Xamarin.Android vNext
  • Universal Windows Platform vNext
Given all of this, the plan for Gepsio is as follows:
  1. Wait for .NET Standard 2.0 to be released. Currently, this is slated to release with the next version of Visual Studio.
  2. Load the current Gepsio code into the next version of Visual Studio.
  3. Migrate the Gepsio projects in Visual Studio away from a simple .NET Framework class library and over to .NET Standard.
  4. Ensure that Gepsio’s XML implementation, both from the standpoint of XML instances and schemas, support .NET Standard 2.0
  5. Cut a release and push out to NuGet.
  6. Continue to move forward with the (admittedly long-delayed) continual improvement of Gepsio’s support for the XBRL standard, now in an environment where several platforms can use the same code base.

Single Assembly Design Reinstated; Stay of Execution for .NET 3.5 Support

The last post discussed a future design whereby Gepsio would ship as two assemblies and that, as a result of this work, would no longer be able to support .NET 3.5.

Never mind.

MissingManifestResourceException Thrown In UWP Calls To PCL Resources

Some work has gone into implementing the design discussed in that last post. The design centered around a Portable Class Library (PCL) which held Gepsio’s string resources as well as a small class that returned strings from the PCL to the caller. In the solution structure, a separate PCL called JeffFerguson.Gepsio.Resources which targeted Windows 10 and .NET 4 was created, and all of the Gepsio assemblies referenced that PCL project. The Gepsio RESX string table was added as a resource in the PCL, and a simple class was added to the PCL:

public class AssemblyResources
{
    public static string GetName(string Key)
    {
        return Gepsio.ResourceManager.GetString(Key);
    }

    public static string BuildMessage(string Key, params object[] Parameters)
    {
        var Message = new StringBuilder();
        string MessageFormat = GetName(Key);
        Message.AppendFormat(MessageFormat, Parameters);
        return Message.ToString();
    }
}

From there, the main Gepsio code would call this PCL class when string resources were needed.

For Gepsio’s .NET assemblies, this all worked well. Gepsio called into the PCL, which accessed a string resource and returned it to Gepsio. Perfect!

For Gepsio’s UWP assembly, however, the same code path crashed. Gepsio called into the PCL, but the call to the Resource Manager’s GetString() method threw an exception:

Result StackTrace:
at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
   at System.Resources.ResourceManager.GetString(String name)
   at JeffFerguson.Gepsio.Resources.AssemblyResources.GetName(String Key)
   at JeffFerguson.Gepsio.XbrlSchema..ctor(XbrlFragment ContainingXbrlFragment, String SchemaFilename, String BaseDirectory)
   at JeffFerguson.Gepsio.XbrlFragment.ReadTaxonomySchemaReference(INode SchemaRefNode)
   at JeffFerguson.Gepsio.XbrlFragment.ReadTaxonomySchemaReferences()
   at JeffFerguson.Gepsio.XbrlFragment..ctor(XbrlDocument ParentDocument, INamespaceManager namespaceManager, INode XbrlRootNode)
   at JeffFerguson.Gepsio.XbrlDocument.Parse(IDocument doc)
   at JeffFerguson.Gepsio.XbrlDocument.Load(String Filename)
   at XbrlConformanceSuiteTests.ConformanceSuiteUnitTests.TestVariation(String testcaseFolderName, String instance, String description, Boolean shouldBeValid)
   at XbrlConformanceSuiteTests.ConformanceSuiteUnitTests.<ExecuteTestcase>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at XbrlConformanceSuiteTests.ConformanceSuiteUnitTests.<XbrlConfCR520120124>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Result Message:	Test method XbrlConformanceSuiteTests.ConformanceSuiteUnitTests.XbrlConfCR520120124 threw exception:
System.Resources.MissingManifestResourceException: Unable to load resources for resource file "JeffFerguson.Gepsio.Resources.Gepsio" in package "bc7a23a2-a1ca-4481-8eb3-74680ae1e9c8".

This issue was brought up in a Stack Overflow question, but no answers were offered.

So much for the PCL being portable.

Resource Management Plan Going Forward

The backup plan is to maintain two separate string tables. One string table will be maintained in the Gepsio .NET projects as an RESX file, and a second string table will be maintained in the Gepsio UWP projects as an RESW file. It’s a shame that the string tables will need to be maintained in two places, but, if a single portable location isn’t going to work, then there is not much choice. Hopefully, it won’t be a huge issue.

Retaining Support for .NET 3.5

The last post mentioned that PCLs do not support .NET 3.5 and, as a result of the proposed Gepsio design making use of PCLs, future versions of Gepsio would not be able to support .NET 3.5. Now that the PCL design has proven to be unworkable, Gepsio can and will support .NET 3.5 for the foreseeable future.

Next Release: Two Assemblies; .NET Support Changes to 4.0 and Above

Gepsio‘s support for the Windows 10 Universal Windows Platform (UWP) will bring two changes to Gepsio that will be new to the platform:
  • Gepsio will ship as two assemblies, rather than a single assembly.
  • Gepsio will no longer be able to support .NET 3.5 and will support only .NET 4.x.
This blog post will examine both of those changes and why they are necessary to be sacrificed to allow Gepsio to support UWP.

A Tale of Two Assemblies

Let’s first take a look at the need for Gepsio to be implemented in two assemblies, rather than just one as in the past.

Gepsio’s Use of String Resources

Gepsio not only parses XBRL, but also validates the contents of an XBRL instance against the XBRL specification. Validation errors are available for inspection after the document instance is loaded, as in the following example:
var myDoc = new XbrlDocument();
myDoc.Load("MyXbrlDocument.xml");
if(myDoc.IsValid == false)
{
    foreach(var currentValidationError in myDoc.ValidationErrors)
    {
        // the currentValidationError.Message property
        // provides more information
        // about the validation error
    }
}
The messages are built into the Gepsio code base through a resource (RESX) file. This RESX file, which is actually an XML document, contains the strings used to build the validation messages. Building these messages into a separate RESX file, rather than embedding them into the source code directly, allows the strings to be translated into various languages by providing a separate XML file for the new language, without having to change the source code itself. Since XBRL is an international specification, it makes sense to keep strings segregated where they can be easily located and changed when Gepsio needs to support languages other than English.
Microsoft changed the way string resources work for UWP assemblies. Resources are now stored in a RESW file, which has the same underlying XML structure as an RESX file but is incompatible with the tooling for .NET applications. Moreover, the code needed to access strings in an RESW file differs from the code needed to access strings in an RESX file.

Possible Design Approaches

The difference in approach between RESX files and RESW files impacts Gepsio, which strives to have a code base that supports .NET as well as UWP. There are several possible approaches to this problem, and they are described below.

One RESX for .NET, One RESW for UWP

With this approach, two resource files would be maintained: an RESX file for the .NET projects, and a RESW for the UWP project. This would cause a maintenance headache, as a developer working on Gepsio would have to remember that any strings added or modified in the RESX would have to be done again in the RESW file, and forgetting to do so will cause an incompatibility between Gepsio’s .NET implementation and its UWP implementation. If a string was found in one file and not the other, Gepsio may even crash at runtime when trying to find a string resource that doesn’t actually exist. Furthermore, since the code needed to access a string from an RESX file differs from the code needed to access a string from an RESW file, the code would also need to be maintained twice. Generally, making two modifications in two files can be more error-prone than making one modification in one file.

One RESX for .NET, Copied And Renamed for UWP

With this approach, one RESX file would be maintained in Gepsio’s .NET project, and a custom step in the build workflow would copy the RESX file over to the UWP project, rename it with an RESW extension, and allow the UWP project to build with the new RESW file. This approach would have the advantage of only needing to maintain one resource file. However, this would also introduce new complexities into the build process and would still require that two separate codebases be maintained: one to access the RESX file and another to access the RESW file.

Resources in a Portable Class Library

With this approach, the project would maintain a Portable Class Library (PCL) containing Gepsio’s string resources. The PCL would be targeted to support .NET as well as UWP. The PCL project would maintain the string resources as part of the project, as a single RESX file in a single location. The PCL would also contain code to access and return strings and Gepsio, in both its .NET and UWP personalities, would call this portable code when strings are needed.

Approach Going Forward

After considering all of the options (and lamenting the fact that the RESX/RESW discrepancy exists in the first place) it was decided to go down the PCL route. This means that, starting with the next release, Gepsio will ship as two assemblies:

  • the main Gepsio assembly
  • the Gepsio string resources assembly

The disadvantage here is that there are now two assemblies to distribute, rather than just the single assembly shipped up this point. The mitigation here is that distribution tools such as NuGet make distribution easier and more automatic, and the chances that a user would ship one Gepsio assembly and forget the other is reduced thanks to these tools.

The other disadvantage to the PCL approach leads us into the next change for the Gepsio platform.

End of Support for .NET 3.5

The “resources in PCL” approach described above signals the end of Gepsio’s support for .NET 3.5 and restricts Gepsio to supporting only .NET 4.0 and above. PCLs do not support .NET 3.5 and their support for .NET goes back only to .NET 4.0. This means that future versions of Gepsio will support the following platforms:
  • .NET 4.0
  • .NET 4.5
  • .NET 4.5.1
  • .NET 4.5.2
  • .NET 4.6
  • .NET 4.6.1
  • Universal Windows
Though .NET 3.5 will be left off of this list, it is most likely not widely used these days. .NET 3.5 was released in November of 2007 and its replacement, .NET 4.0, was released in April of 2010. It is therefore possible than any installations using .NET 3.5 at one time have been upgraded some time in the last six years to one of the .NET 4.x runtimes.

No Semantics to Context References

This email came into the Gepsio inbox recently:

I’ve been playing around with the library. My first exercise for myself is trying to reconstruct the financial statements. I thought I’d group Facts by time periods. The ContextRefNames were not what I expected. There were overlaps in names like FD2015XXX, D2015XXX, I2015XXX. Do you know the difference between the prefixes? I’m using Whole Foods https://www.sec.gov/Archives/edgar/data/865436/000086543615000177/0000865436-15-000177-index.htm

Is there a particular convention that the contextrefnames follow? I had a look at Walmart and they use a similar, but different convention.

Unfortunately, the answer is “no”.

Context IDs and references have no meaning. They are syntax, not semantics. Context reference names are arbitrary and have meaning only to link items within the individual document instances.

Section 4.7.1 of the XBRL 2.1 specification discusses the “id” attribute of a <context> element:

Every  <context>  element MUST include the  @id attribute. The content of the  @id attribute MUST conform to the [XML] rules for attributes with the ID type (http://www.w3.org/TR/REC-xml#NT-TokenizedType). The  @id attribute identifies the context (see  Section 4.7) so that it may be referenced by item elements.

The specification does not impose any semantics on the contents of the ID attribute.

Gepsio for Universal Windows Underway

The following email came in to the Gepsio inbox today:

Hi,

I am very interested in trying out Gepsio on the windows 10 universal app platform.

I saw that a universal windows project has been added to the repo without some vital xml parts.

Will there be a new patch soon with the remaining parts?

As a matter of fact, yes! That work is underway now.

The email is referencing this changeset, which contains the following somewhat-cryptic description:

Universal Windows project added. XML implementation for Universal Windows is not implemented, and there are no unit tests for Universal Windows, so Gepsio for Universal Windows cannot be used at this time. The .NET builds are, as always, fully supported.

That changeset included a new project in the Gepsio solution – a project which would target a Gepsio assembly tuned for the Windows 10 Universal Windows Platform (or UWP, for short). The vision is to allow Gepsio to be used by developers wishing to build XBRL-enabled applications on Windows 10 UWP.

Thanks to the interface-based separation between Gepsio’s XML service layer and its XBRL semantic layer, described here, the work needed to allow Gepsio to support Windows 10 UWP is not a “do over”. The work involved is, basically, to build new XML interface implementations that use Windows 10 UWP instead of .NET. This work is underway now. In fact, Gepsio already has enough UWP code to open up an XML document and search for <xbrl> nodes in Windows 10. This is exciting, as it means that Gepsio can be used for code that runs on any Windows 10 UWP platform, from the phone, to the tablet, to the desktop, to the Xbox One.

The Windows 10 UWP code is not ready, and, although the project skeleton has been checked in, the XML layer for Windows 10 UWP is not ready. That code is in progress. When it’s ready, there will be a blog post announcing the new code. The code is not a straightforward port from the .NET XML code, for the following reasons:

  • The .NET code uses the XML classes in the System.Xml namespace. The Windows 10 UWP code is using XML classes in the Windows.Data.Xml.Dom namespace.
  • The Windows 10 UWP makes use of asynchronous methods; for example, XmlDocument.Load() has become await XmlDocument.LoadFromFileAsync(). Ensuring that Gepsio supports this asynchronous model, while still maintaining the existing needs of the Gepsio XBRL semantic layer, will need some additional work.
  • Unlike .NET’s System.Xml namespace, the Windows 10 UWP’s Windows.Data.Xml.Dom namespace does not have any built in support for XML Schema. XBRL relies on XML Schema documents for its taxonomies. Gepsio’s XML Schema support will have to be written from the ground up since there is no support for XML Schema in UWP.

Gepsio will retain its .NET builds as well. Gepsio will continue to support .NET 3.5, .NET 4, .NET 4.5, .NET 4.5.1, .NET 4.5.2, and .NET 4.6. If you’re not ready to move to Windows 10 UWP just yet, do not worry – all of the .NET builds will continue to be supported and ship for the foreseeable future.