NuGet Team Blog

Issues uploading portable library packages [Fixed]

Starting September 3rd, some users started having issues uploading packages that contain portable libraries; on September 10th, we deployed a fix to address those issues.

Errors Experienced

Depending on how you were publishing your package, you would have experienced the errors in a few different ways.

Upload Package page on nuget.org

If you were uploading your package through the nuget.org website’s Upload Package page, the package would have initially uploaded okay, but you would have gotten an error when submitting the verified package details.

The error would have been our standard “Error: Oh no, we broke something!” response page.

Push with nuget.exe

If you were using nuget.exe push to upload your package to the gallery, you would have gotten a different, and misleading error. That error was coming back as “The remote server returned an error: (502) Bad Gateway.”

Publish through NuGet Package Explorer

NuGet Package Explorer has the ability to publish packages directly. Attempting to publish packages with portable dependencies would lead to a vague error about a “bad request.”

Broken Scenarios

It took us a few days to really get to the root cause of this issue, because we were observing a couple of different behaviors. When we finally had it figured out, here’s what it turned out to be.

Scenario 1: Packages with portable libraries

In the scenario of a package containing portable libraries, there were a few details necessary to hit the error.

  1. The \lib folder of the package contained a subfolder for a portable library profile. Example: \lib\portable-net4+sl4+wp71+win8
  2. The nupkg file was created using nuget.exe pack.
  3. The nupkg file was being uploaded either through the Upload Package page on the gallery, or from nuget.exe push.

If the nupkg file was created with NuGet Package Explorer, or just opened and saved in it, then the package could be uploaded successfully. If the nupkg file was published to the gallery directly from NuGet Package Explorer, that would also succeed.

What we learned from this scenario was that even though NuGet Package Explorer uses the same endpoint as nuget.exe push, packages created within it were immune to the error.

Scenario 2: Packages with portable dependencies

Some package authors were bypassing scenario 1 by using NuGet Package Explorer to create their package, but they were still getting the errors. In this scenario, we found the packages had dependencies that only applied for portable libraries. For instance, a dependency on Microsoft.Bcl.Async which only applies for portable library projects. NuGet Package Explorer would save this dependency’s target framework as “.NETPortable,Version=v0.0,Profile=net45+sl4+sp71” and that would trigger the errors during package uploads as well, even when publishing through NuGet Package Explorer.

Root Cause

The root cause of this issue ended up being a little complicated. Without diving too much into the details, here’s the general explanation, broken down into a few steps.

Step 1 - Creating the package

  1. Portable library folders names include + characters
  2. nuget.exe (built against .NET 4.0) encodes the + characters, changing them to %2B
  3. NuGet Package Explorer (built against .NET 4.5) does not encode the + characters

Step 2 - Uploading the package

  1. When processing a package upload, nuget.org always decodes folder names for package contents to determine the package’s supported frameworks
  2. When the package was created with nuget.exe, the %2B was decoded back to a + character
  3. When the package was created or published with NuGet Package Explorer, the + character was decoded into a space
  4. The decoded folder names determine the list of frameworks for the package

Step 3 - Verifying target frameworks

  1. With the list of frameworks, nuget.org verifies frameworks before storing them in the list of supported frameworks for the package
  2. If unrecognized characters (such as spaces) are found in the framework name, the framework name is thrown out immediately
  3. If the framework name contains only valid characters, and it’s found to be a portable library, we load the portable libraries from the .NET Framework installation and parse them to create normalized framework names

Outcome

We found at this stage of the process that there were two possible outcomes:

  1. Packages created or published with NuGet Package Explorer, whose decoded folder names had spaces in them, would get ignored as invalid framework names – this was causing data corruption where we weren’t capturing the portable profiles supported by the package
  2. All other packages, whose decoded folder names had the correct + characters, would fail to parse because of a bug in how we were reading the normalized framework names from the .NET Framework

Note that packages in Scenario 2 above bypass the folder name encoding/decoding and they were skipping directly to outcome number 2 where the framework name parsing would fail.

For the first outcome, we handled the exception gracefully but this led to data corruption–it was actually a silent failure.

For the second outcome, we were not handling the exception and it was causing the visible errors users reported.

The Fix

We needed a two-part fix to address the issues here.

  1. Parse the framework names correctly so that we don’t fail trying to read the normalized framework name from the .NET Framework.
  2. Prevent spaces in the decoded folder names by putting the + characters back in place.

On September 10th, we deployed these fixes to the gallery and portable library packages can again be uploaded (with no data corruption). We are still working on correcting the data behind the packages that were successfully published over the past week that incurred the data corruption. The symptom of the data corruption is that despite the package being available on the gallery, users with previous versions of the package won’t always see the new version as an available update.

Our Apologies

We are sorry for the trouble this issue has caused package authors over the past week. We appreciate that you immediately reached out to us and that you were patient while we investigated and fixed the bugs.

Published September 10, 2013

NuGet 2.7 Package Restore Consent Errors

After NuGet 2.7 was released with Automatic Package Restore and implicit consent, some users reported that they were still seeing build errors indicating that package restore consent had not been given. When we made the package restore changes in NuGet 2.7, we identified one scenario where this would happen but determined we couldn’t implement a fix and would rather have to document the cause and solution.

Scenario and Cause

In order to hit this problem, there’s a specific scenario you must be in. The scenario isn’t extremely common, but we have already heard reports of it.

  1. Using a freshly built machine or a machine where Visual Studio was freshly installed
  2. NuGet has been upgraded to NuGet 2.7
  3. An existing solution is opened where the MSBuild-integrated package restore has already been enabled
  4. The MSBuild-integrated package restore was enabled before NuGet 2.7 was released

In this scenario, the solution contains a .nuget folder that has a nuget.exe in it versioned 2.6 or earlier. In those versions of nuget.exe, package restore consent was OFF by default (hard-coded to false when not present in nuget.config). When building this solution in Visual Studio, NuGet identifies that the MSBuild-integrated package restore is enabled and therefore automatic package restore is skipped. Then the old nuget.exe kicks in for the package restore and it does not find package restore consent to be given, yielding the following build error:

Package restore is disabled by default. To give consent, open the Visual Studio Options dialog, click on Package Manager node and check ‘Allow NuGet to download missing packages during build.’ You can also give consent by setting the environment variable ‘EnableNuGetPackageRestore’ to ‘true’.

Solution

As explained on the Package Restore documentation, there are three ways to address this situation.

  1. Force save your NuGet settings with consent given. To do this, open Visual Studio’s options and under Package Manager, choose General. Uncheck and then re-check the boxes for consent and click OK. This forces your %AppData%\NuGet\NuGet.config file to be saved with consent explicitly given, allowing NuGet 2.6 and earlier to see that you’ve given consent.
  2. Update the version of NuGet.exe in your .nuget folder. To do this, run nuget.exe update -self from your .nuget folder, which will download the latest version of NuGet.exe and replace the version in the .nuget folder. The latest version of NuGet.exe will infer consent to be ON even when not explicitly saved in the NuGet.config file.
  3. Migrate to Automatic Package Restore. For this approach, you would migrate from the MSBuild-integrated package restore to the Automatic Package Restore approach, following the documented walkthrough.

Published August 29, 2013

Editing Package Metadata on the NuGet Gallery

In our next deployment, which is scheduled for the week of August 26th, 2013, we have added the ability to edit package metadata at any point in time before or after publishing the package. This has been one of the frequent requests from package authors.

The changes are current dpeloyed @ staging.nuget.org. You can play around with the feature in this sandbox environment. Let us know your feedback and suggestions here in github. We are aiming production deployment on 8/27/2013 barring any ship-stopper issues. Meanwhile, we will continue testing from our end as well.

Edit Package

Before the introduction of this feature, clicking on “Edit package” on your package page will give you the below message.

Old EditPackage

With this new feature, the “Edit package” page will display all of the metadata of the package and lets you to update them.

All the descriptive fields of the package like Title, Description, Summary, Icon Url, Project Url, Authors, Copyright, Tags and Release notes can be edited. However, updates to the functional details of the package like ID,Version, Dependencies and actual the content of the package requires a new version to be uploaded. This is inline with our current behavior. More details can be found in the spec and the forum discussion thread.

Editing different versions of a package

You can chose any specific version of the package (not necessarily the latest) and edit its metadata. The “Edit package” page lists the package versions in the left pane making it easier to switch between versions.

Edit Package Versions

Saving and undoing edits

Once you update the desired fields of the package and “Save” the edit, the changes will be submitted. It might take a few minutes for the new nupkg file to be created and the changes to reflect in the web page. When the changes are being applied, you (as the package owner) will see the below message in the package page and the new updated data, whereas other users would still see the old data.

Pending Edit Message

You can click on “Undo pending Edits” link in the notification message to undo all the current edits before it gets applied. You can always make another edit at a later point of time to make further changes.

Editing while uploading a new package

Package details can be edited while uploading a new package as well. The “Verify Details” page now displays all of the metadata of the package. You can update the details as required and a nupkg file with the modified data will be created and uploaded for you. This will help you in making quick edits to the package before publishing it.

Edit Package On Upload

Search/Install/Update on edited package

Once the pending edit goes through, all search/Install/update operations on the package from VS client or the api/v2 feed will reflect the new data and nupkg.

Note, it might take up to 10 minutes for the new changes to be reflected in search results (as the index needs to be updated).

How it Works

When a package author submits a package, we queue the edit up for being applied. Our backend system processes the queue of edits the order they were submitted, creating an updated NuGet package file (.nupkg), replacing the originally uploaded package and leaving the version number the same. The package originally uploaded, along with its corresponding metadata, are archived in the system.

As a result of the newly created nupkg file, the search index and package source feed are updated as well. This will have downstream effects on consumers of the package source feed as the package’s metadata and hash will be updated. From that point forward, users that install the package will get the updated package, while users that installed the package before the edit will obviously have the old package. When package restore is used, the updated package will be downloaded during package restore, seamlessly replacing the original.

The end result is that package authors now have the ability to update package metadata without publishing a new version of the package. We hope this approach will address the many scenarios when package authors have needed the ability, while also not disturbing package consumers in any way.

Published August 23, 2013

Older Posts

More posts can be found in the full archive.