Migrate data management for a Nextstrain Group from S3 to nextstrain.orgď
Hint
This guide only applied to Nextstrain Groups created before March 2022.
As of June 2023, all such groups have been migrated. This guide is preserved only as a record.
Whatď
Weâve improved the way to manage datasets and narratives in Nextstrain
Groups. Now the Nextstrain CLI will directly interact
with nextstrain.org instead of interacting with a group-associated AWS S3
bucket (e.g. s3://nextstrain-${GROUPNAME}/
).
For comparison, hereâs an example of the old way vs. the new way to upload a dataset to a Nextstrain Group:
# Old way
nextstrain remote upload \
s3://nextstrain-${GROUPNAME}/ \
auspice/${YOUR_BUILD_NAME}.json \
auspice/${YOUR_BUILD_NAME}_tip-frequencies.json \
auspice/${YOUR_BUILD_NAME}_root-sequence.json
# New way
nextstrain remote upload \
nextstrain.org/groups/${GROUPNAME} \
auspice/${YOUR_BUILD_NAME}.json \
auspice/${YOUR_BUILD_NAME}_tip-frequencies.json \
auspice/${YOUR_BUILD_NAME}_root-sequence.json
Each group must undergo a short migration (performed by us) in order to use the new way. Groups which havenât been migrated will continue to use the old way.
With the new way, group data is still stored in S3 with access controlled by Nextstrain, but now you no longer need to setup or pass around AWS credentials provided to you by Nextstrain. Instead, you will login using the same nextstrain.org username and password you use to login to the website. Each CLI login session is valid for at most 30 days, after which youâll need to login again. Commands requiring login will prompt you to do so when necessary.
Whyď
This change simplifies the management of Nextstrain Groups for both users and ourselves, primarily by removing the need to setup and keep track of a separate set of AWS credentials for each group. It also makes it easier for us to provision new groups, which is an important step in making Nextstrain Groups more widely available.
The new membership roles are made possible by the change, including the possibility of finer-grained roles/permissions as necessary in the future.
Actions requestedď
Each group is migrated independently of any others. To make your groupâs migration as smooth as possible, weâre requesting some help from you.
Before migrationď
Pick a date and/or time of day youâd like the migration to happen.
During the brief migration period (less than an hour), you wonât be able to manage your groupâs data. However, youâll be able to view the datasets and narratives in your group on nextstrain.org the whole time, without any downtime.
If you have no preference, you may leave it up to us to pick a date and time.
Decide on roles for existing group members.
Groups now support three membership roles (viewers, editors, and owners) which determine who can manage data.
By default, all existing members will be viewers unless you tell us otherwise.
Request that we migrate your group by emailing us.
Include your preferred date and/or time of day and the list of roles for existing members. Weâll confirm the date and time of your migration in advance of performing it.
Note
If you frequently update your groupâs overview or logo files, please email us to let us know and wait to migrate until weâve added support for updating these files yourself when not using a separate S3 bucket.
Upgrade the version of the Nextstrain CLI youâre using to at least 5.0.0.
Itâs best to do this in advance of the migration so youâre set to keep using your group afterwards without having to upgrade later, but you may choose to wait.
Check the version you have by running:
$ nextstrain version nextstrain.cli 5.0.0
If you see a version older than 5.0.0, please upgrade your copy of the Nextstrain CLI.
After migrationď
Weâll email you when your groupâs migration is complete. From that point forward, everyone managing your groupâs datasets and narratives will need to use ânextstrain remoteâ commands which reference your groupâs nextstrain.org URL instead of your previous S3 bucket URL. You will no longer have access to your previous S3 bucket, and after a grace period of at least 30 days, it will be completely deleted.
Itâs a good idea to give these new commands a try shortly after the migration
to make sure everything works as expected for you. For example, you might try
logging in and listing your groupâs datasets. Hereâs what that looks like for
the user trs
and the group blab
:
$ nextstrain login
Logging into Nextstrain.orgâŚ
Username: trs
Password:
Credentials saved to /home/tom/.nextstrain/secrets.
Logged into nextstrain.org as trs.
Log out with `nextstrain logout`.
$ nextstrain remote list groups/blab
https://nextstrain.org/groups/blab/Pf/K13
https://nextstrain.org/groups/blab/Pf/chr13
https://nextstrain.org/groups/blab/beta-cov
https://nextstrain.org/groups/blab/ncov/19B
âŚ
Youâll of course have to login as yourself, and be sure to replace blab
with your groupâs name when you try it!
If you need to make changes to the roles of any group members or update your groupâs overview or logo file, email us and weâll take care of it. In the future, group owners will be able to make these changes themselves.
Timelineď
- March 2022
New groups created after this point manage their data through nextstrain.org instead of S3.
- early November
Notification sent to groups created prior to March.
- November onwards
Groups start migrating one-by-one in coordination with group owners.
- end of February 2023
Nextstrain teamâs desired deadline for migrating all groups.
- June 2023
All affected groups have been migrated.
Notificationď
In November 2022, we emailed the contacts for all groups created before March 2022 to inform them of these changes. A copy of the email is below.
Subject: Action requested: Improvements to how you manage your Nextstrain Group
Hello!
Itâs the Nextstrain team. Weâre writing to let you know about improvements weâve made to how you manage your data in Nextstrain Groups. Weâve put together a documentation page with information about the changes, including the actions requested of you. Please review the details there and reply to this email with the information requested.
Youâre receiving this email because youâre an owner of one or more Nextstrain Groups:
nextstrain.org/groups/${GROUPNAME}
Thank you for being an early adopter of Nextstrain Groups! Weâre excited to keep improving the functionality of Groups. If you have any feedback, please donât hesitate to email us or post to discussions.nextstrain.org.
âthe Nextstrain team
On 10 January 2023, we emailed a reminder to the contacts for groups not yet migrated.
Subject: Action required: Upgrade of your Nextstrain Group
Hello!
Itâs the Nextstrain team. We previously emailed you on 7 November 2022 about improvements weâve made to how Nextstrain Groups are managed. This is a reminder since we havenât heard back from you yet and would like your co-operation when upgrading your Groups.
If we donât hear back from you by the end of February 2023, we will proceed with upgrading your Groups without your involvement. Upgrading without your involvement will disrupt your ability to upload new datasets and manage existing datasets until you complete your own part of the upgrade.
Weâve put together a documentation page with information about the changes, including the actions required of you. Please review the details there and reply to this email with the information required before the end of February 2023.
Youâre receiving this email because youâre an owner of one or more Nextstrain Groups:
nextstrain.org/groups/${GROUPNAME}
Thank you for being an early adopter of Nextstrain Groups! Weâre excited to keep improving the functionality of Groups. If you have any feedback, please donât hesitate to email us or post to discussions.nextstrain.org.
âthe Nextstrain team
On 15 February 2023, a final reminder email was sent to the contacts of several remaining groups.
On 10 March 2023, the bulk of the several remaining groups were migrated without the involvement of their contacts, and they were notified via email.