| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-21 | |||
| 20:30:17 | imacdonn | but back to the checks ... I think it may be a good idea, especially if you're trying to make this a generic framework, so have places for pre- and post-upgrade checking | |
| 20:30:22 | mriedem | which isn't going to be how everyone does it | |
| 20:30:37 | mriedem | well, pre and post upgrade are pretty fuzzy, | |
| 20:30:54 | mriedem | the general rule i try to follow is how idempotent can i write the check such that anyone can run it at anytime | |
| 20:31:17 | mriedem | be that before the db sync with new code, after that but before new code starts running, after new code is running, etc | |
| 20:31:26 | mriedem | i know that doesn't help you much here... | |
| 20:31:54 | mriedem | it makes more sense when you have something you actually need to write a check for, | |
| 20:32:21 | mriedem | e.g. if your release notes say, "make sure you do x before upgrading to stein" then that's a pretty obvious thing to check (if you can) | |
| 20:32:38 | imacdonn | I can sortof see that ... but it seems that what constitutes "OK to start the upgrade" vs. "the upgrade did everything it was supposed to, and you can start services now" may be different | |
| 20:38:03 | imacdonn | Looking at the output of 'nova-status upgrade check', I see one of the items is "Request Spec Migration" | |
| 20:38:47 | imacdonn | haven't looked to see exactly what that is, but if it's a migration, and migrations don't happen during sync, and you're supposed to do the upgrade check before running the new code .............. | |
| 20:39:50 | mriedem | so the story on that one is, | |
| 20:39:59 | mriedem | nova added this thing called a request spec in newton, | |
| 20:40:13 | mriedem | and has compatibility code all over the place for instances created before newton that don't have a related request spec record in the db, | |
| 20:40:27 | mriedem | we have an online data migration that operators can run to create records for old instances, | |
| 20:40:41 | mriedem | the upgrade check was added because i want to drop that compatibility code in stein, | |
| 20:41:16 | mriedem | which means, before you get to stein and restart services with stein code (which could have dropped that compatibility), you can run that check and see, oh i better finish my homework - it's been due for 2 years | |
| 20:42:54 | imacdonn | so ... if we're running this check after doing the db sync with the stein code, but before we can start up the services with the new code, can we still do the online migration ? | |
| 20:43:07 | mriedem | so the way upgrades work in grenade for nova is: lay down new code, sync the db schema for the new models, run the online data migrations, restart the services | |
| 20:43:19 | mriedem | *with nova-status upgrade check in between those last 2 | |
| 20:43:31 | mriedem | sure you can | |
| 20:43:56 | mriedem | the online data migrations CLI in nova just needs access to the config and db | |
| 20:44:02 | mriedem | none of that goes over rpc or rest api | |
| 20:44:12 | mriedem | otherwise you can't FFU | |
| 20:45:35 | imacdonn | so "online" means that it's OK to do online, not that it has to be done online? or am I completely misunderstanding what "online" means ? | |
| 20:46:58 | mriedem | the 'online_data_migrations' name might be a bad name | |
| 20:47:08 | mriedem | it just means, | |
| 20:47:16 | mriedem | not offline like during a db sync when the controller is down | |
| 20:47:24 | mriedem | like how data migrations were done during db sync in the long ago | |
| 20:47:34 | mriedem | which meant operators could have substantial downtime in the control plane | |
| 20:48:39 | imacdonn | ok | |
| 20:49:28 | mriedem | which project are you looking to do the upgrade checks for btw? | |
| 20:49:40 | imacdonn | so I need to digest the whole upgrade flow a bit more (maybe after I digest lunch) .... but I still think it's worth at least considering pre/post-upgrade checks in general | |
| 20:50:21 | imacdonn | I'm trying to be a good citizen and 'do the right thing' for all projects .. or all of the ones that I uses, which is mostly the core ones | |
| 20:51:40 | imacdonn | based on your ML posts a while back, I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services | |
| 20:52:15 | imacdonn | maybe mostly because currently placement is considered a nova service, at least in my deployment mechanism | |
| 20:53:03 | imacdonn | I may try hacking in sometihng to start placement individually before doing the upgrade check, then start the rest of the (nova) services | |
| 20:53:47 | imacdonn | or maybe I'll worry about that for stein | |
| 20:54:47 | mriedem | "I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services" - as i said, that's also acceptable in the case of nova and it's even in our install docs: https://docs.openstack.org/nova/latest/install/verify.html | |
| 20:54:55 | mriedem | for a fresh install i mean | |
| 20:55:29 | mriedem | flexibility and idempotency are kind of the key things here | |
| 20:55:54 | imacdonn | Well, the description does say "Performs a release-specific readiness check before restarting services with new code" | |
| 20:56:01 | imacdonn | https://docs.openstack.org/nova/latest/cli/nova-status.html | |
| 20:56:39 | imacdonn | but yeah ... I was basically using it as a validation that "everything went OK with the nova upgrade" | |
| 20:56:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423 | |
| 20:58:14 | imacdonn | re fresh install vs upgrade, I basically perform the same steps in both cases (that idempotence thing) | |
| 21:01:54 | mriedem | yeah some of the wording is a bit dated and was written when the initial change in ocata was really for before the new code was started up | |
| 21:02:12 | mriedem | but that's still how we do it in grenade | |
| 21:03:36 | imacdonn | wondering if there might be a case where you really do want to check for something before starting the new code, because otherwise the new code might try to do something that would turn out to be catastrophic | |
| 21:04:15 | mriedem | well, yes, you want to run it after starting new code | |
| 21:05:17 | mriedem | guh, sorry, typo | |
| 21:05:21 | mriedem | before starting new code | |
| 21:05:22 | mriedem | that's the idea | |
| 21:05:33 | mriedem | put it in a venv or container and run it | |
| 21:47:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add some more docs for upgrade checkers https://review.openstack.org/604486 | |
| 22:26:28 | imacdonn | mriedem_away: On a slightly related note; is there a reason that this do like .where(compute_nodes.c.deleted==0) ? https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L111-L130 | |
| 22:29:43 | imacdonn | erm, I mean; ... that this doesn't* do like ..... | |
| 22:32:41 | imacdonn | see also https://bugs.launchpad.net/nova/+bug/1757207 | |
| 22:32:41 | openstack | Launchpad bug 1757207 in OpenStack Compute (nova) "compute resource providers not equal to compute nodes in deployment" [Undecided,Incomplete] | |
| #openstack-nova - 2018-09-22 | |||
| 00:10:24 | karimull | Hi mriedem : a qk question , can we add plugins in nova-compute , which can be called before libvirt is executed for special processing during instance launch? | |
| 00:59:18 | openstackgerrit | iain MacDonnell proposed openstack/nova master: nova-status - don't count deleted compute_nodes https://review.openstack.org/604495 | |
| 01:19:05 | openstackgerrit | Merged openstack/nova stable/rocky: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604367 | |
| 04:30:36 | openstackgerrit | Merged openstack/nova master: remove virt driver requires_allocation_refresh https://review.openstack.org/602481 | |
| 04:30:43 | openstackgerrit | Merged openstack/nova master: Nix update_instance_allocation, _allocate_for_instance https://review.openstack.org/604186 | |
| 06:50:21 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260 | |
| 11:29:17 | openstackgerrit | Brin Zhang proposed openstack/nova master: Specifies the storage backend to boot instance https://review.openstack.org/579360 | |
| 16:06:25 | mriedem | imacdonn: replied | |
| 16:07:03 | mriedem | karimull: there are "hooks" so you could write something out of tree to run before/after the driver spawn in the compute service, however, hooks are deprecated and your out of tree code could be broken at any time | |
| 16:07:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/hooks.py | |
| 16:24:52 | imacdonn | mriedem: thanks! | |
| 16:30:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/604376 | |
| 17:20:41 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226 | |
| 19:57:06 | openstackgerrit | Merged openstack/nova stable/rocky: Fix service list for disabled compute using MC driver https://review.openstack.org/592310 | |
| 20:32:29 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/pike: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/604039 | |
| #openstack-nova - 2018-09-23 | |||
| 05:17:24 | openstackgerrit | Merged openstack/nova master: Allow ability for non admin users to use all filters on server list. https://review.openstack.org/526558 | |
| 06:47:15 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260 | |
| 06:53:37 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/604577 | |
| 08:08:05 | openstackgerrit | Brin Zhang proposed openstack/nova master: Specifies the storage backend to boot instance https://review.openstack.org/579360 | |
| 10:04:17 | openstackgerrit | Brin Zhang proposed openstack/nova master: Rename "polling_changes-since_parameter.rst" https://review.openstack.org/604606 | |
| 10:43:24 | openstackgerrit | Brin Zhang proposed openstack/nova master: Rename "polling_changes-since_parameter.rst" https://review.openstack.org/604606 | |
| 10:52:09 | openstackgerrit | Hamdy Khader proposed openstack/nova master: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/602351 | |
| 15:01:18 | openstackgerrit | melissaml proposed openstack/python-novaclient master: Update the URL in doc https://review.openstack.org/604629 | |
| 15:28:55 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 15:48:16 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| #openstack-nova - 2018-09-24 | |||
| 02:15:15 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 06:23:18 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add volume_type field to BlockDeviceMapping object https://review.openstack.org/604687 | |
| 06:57:44 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add volume_type field to BlockDeviceMapping object https://review.openstack.org/604687 | |
| 06:58:55 | gibi | mriedem: thanks for the review, I'm planning to focus on that patch series this week | |
| 06:59:34 | gibi | cdent: bedisdes the use-nested-allocation-candidates I think bauzas working on the vgpu support that will utilize nested scheduling too | |
| 07:05:16 | gibi | s/bedisdes/besides/ :) | |
| 08:03:09 | openstackgerrit | Lee Yarwood proposed openstack/nova master: placement: Always reset conf.CONF when starting the wsgi app https://review.openstack.org/604693 | |
| 08:03:47 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: placement: Always reset conf.CONF when starting the wsgi app https://review.openstack.org/604694 | |
| 08:31:37 | bauzas | good morning nova | |
| 08:31:41 | bauzas | I wave a bit late tho | |
| 08:46:15 | gibi | bauzas: good morning | |
| 08:52:09 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226 | |
| 09:05:55 | zioproto | hello | |
| 09:06:12 | zioproto | I am upgrading my cluster from Newton to Ocata, and I see weird failures in Nova like this http://paste.openstack.org/show/730608/ | |
| 09:06:24 | zioproto | does anyone had such problems with database foreign keys ? | |
| 09:06:29 | zioproto | this seems to be related to heat | |