| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 20:48:35 | melwitt | +2, looks cool to me | |
| 20:49:41 | melwitt | thanks for changing those test names, makes it a lot clearer to someone not in-the-know | |
| 20:51:39 | edleafe | melwitt: yeah, well, they went through a bunch of back-and-forth as people had different opinions on how the migration should work. | |
| 20:57:47 | openstackgerrit | Merged openstack/nova master: update policy UT fixtures https://review.openstack.org/398610 | |
| 20:58:28 | openstackgerrit | Merged openstack/nova master: Require Placement 1.10 in nova-status upgrade check https://review.openstack.org/492234 | |
| 20:59:09 | openstackgerrit | Merged openstack/nova master: Add For Operators section to front page https://review.openstack.org/491815 | |
| 20:59:51 | mikal | mriedem: replying now | |
| 20:59:53 | openstackgerrit | Merged openstack/nova master: rework index intro to describe nova https://review.openstack.org/491834 | |
| 21:00:37 | openstackgerrit | Merged openstack/nova master: Bulk import all config reference figures https://review.openstack.org/492105 | |
| 21:01:20 | mriedem | if i work from home, and your dog next to my house barks non-stop for 30+ minutes, | |
| 21:01:23 | mriedem | i should be able to do something bad | |
| 21:02:06 | openstackgerrit | Merged openstack/nova master: nova-manage: Deprecate '--version' parameters https://review.openstack.org/453808 | |
| 21:04:02 | openstackgerrit | Merged openstack/nova master: doc: Import configuration reference https://review.openstack.org/491853 | |
| 21:04:45 | openstackgerrit | Merged openstack/nova master: Structure cli page https://review.openstack.org/492111 | |
| 21:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: address review comments in stable-api guide updates https://review.openstack.org/492690 | |
| 21:12:40 | mtreinish | efried: hmm, the patch would have fixed the 502 errors. It likely means something is misconfigured elsewhere in the path | |
| 21:12:45 | mtreinish | let me take a look at the gate logs | |
| 21:13:18 | bauzas | mriedem: le woof says hello to your neighbor :p | |
| 21:13:33 | mriedem | god | |
| 21:13:44 | mriedem | i hope le woof isn't as dumb as the neighbor dog | |
| 21:13:54 | openstackgerrit | Robert Ellis proposed openstack/nova master: Clarifying node_uuid usage in ironic driver. https://review.openstack.org/485803 | |
| 21:14:56 | bauzas | eurasier FTW | |
| 21:15:39 | bauzas | FWIW https://review.openstack.org/#/c/487954 looks good to me, but is the Ironic job working fine ? | |
| 21:16:21 | mriedem | this is just an uncut newfoundland | |
| 21:17:24 | mriedem | the last ironic job run failed http://logs.openstack.org/54/487954/13/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/fa780de/console.html | |
| 21:17:28 | mriedem | looks like due to timeout | |
| 21:17:54 | mriedem | PS12 was ok http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/ | |
| 21:18:04 | mriedem | PS14 was citycloud-lon1 which is a known slow node issue right now | |
| 21:18:33 | bauzas | how many times is running the refresh cache? | |
| 21:18:41 | bauzas | I mean the period | |
| 21:21:40 | mriedem | bauzas: well, at least every 60 seconds by default because it's called from get_available_nodes which is called from the update_available_resources periodic task | |
| 21:23:10 | bauzas | I'm trying to understand if concurrent runs would be some problems | |
| 21:23:16 | bauzas | looks not | |
| 21:23:49 | bauzas | that said, a question | |
| 21:24:05 | bauzas | if we spawn, do we have the same cache ? | |
| 21:24:08 | bauzas | mriedem: ^ | |
| 21:24:49 | mriedem | i don't understand the question | |
| 21:25:40 | bauzas | we would run multiple greenlets, right? | |
| 21:25:54 | bauzas | so my question is about the node cache | |
| 21:26:06 | bauzas | do we share the same node cache object between greenlets ? | |
| 21:26:55 | mriedem | oh eventlet spawn | |
| 21:27:00 | mriedem | not driver.spawn | |
| 21:28:21 | bauzas | yup eventlet.spawn_n even | |
| 21:28:59 | bauzas | maybe it's a stupid question, but I'm not remembering if coroutines accept to just share the same objects | |
| 21:29:59 | mriedem | _pike_flavor_migration is passed to the spawn and updates self._migrated_instance_uuids and yes i'd assume that's all pointing back to self | |
| 21:30:01 | mriedem | as the same object | |
| 21:30:10 | mriedem | otherwise that would be crazy | |
| 21:32:03 | mriedem | http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/logs/screen-n-cpu.txt.gz#_Aug_09_19_31_21_252127 | |
| 21:32:03 | mriedem | you see things getting hit in the logs | |
| 21:32:11 | mriedem | because the ironic jobs don't yet set resource_class on the nodes | |
| 21:32:37 | bauzas | mriedem: I should explain more my concern | |
| 21:32:50 | bauzas | mriedem: say we have pike flavor migration run that takes more than 60 secs | |
| 21:33:03 | bauzas | then, we would have 2 concurrent migrations | |
| 21:33:13 | bauzas | that looks okay to me because we check the cache | |
| 21:33:53 | bauzas | but the question I wonder is whether all the greenlets share the same cache, so when the first migration updates the cache, the latter gets the updates | |
| 21:34:06 | bauzas | maybe it's pointless, and I'm just silly | |
| 21:34:17 | bauzas | but I'm just thinking out loud | |
| 21:34:51 | mriedem | both greenlets should be working on the same self._migrated_instance_uuids | |
| 21:34:56 | mriedem | melwitt: edleafe: ^? | |
| 21:35:44 | bauzas | mriedem: looking at StackOverflow, looks like yup | |
| 21:35:47 | melwitt | yeah, I think the self._migrated_instance_uuids would be shared between them (if one was still running longer than 60 sec). so maybe we need to synchronize access to that set? | |
| 21:36:20 | mriedem | oh if only we could be using synchronized collections from java! | |
| 21:37:34 | mriedem | what's the worst that would happen here? | |
| 21:37:47 | mriedem | wouldn't we just double migrate the same instance.flavor.extra_spec? | |
| 21:37:52 | melwitt | that's what I was trying to think about. | |
| 21:38:03 | mriedem | continue | |
| 21:38:03 | mriedem | # has already been migrated | |
| 21:38:03 | mriedem | # The compute must have been restarted, and the instance.flavor | |
| 21:38:03 | mriedem | if resource_key in specs: | |
| 21:38:06 | bauzas | mriedem: melwitt: well, the more I think about the problem, the more I think it wouldn't be a prolem | |
| 21:38:06 | mriedem | ^ should save it | |
| 21:38:16 | bauzas | at least because it's for flavors | |
| 21:38:20 | melwitt | yeah | |
| 21:38:33 | bauzas | not sure operators have a lot of flavors needing more than 60 secs for a migration | |
| 21:39:17 | mriedem | well, unless you're hitting rpc timeouts on sending updates to conductor or something | |
| 21:39:26 | bauzas | and if so, well, not a problem given the current code which is not synchronised but failproof | |
| 21:39:39 | melwitt | it's a good point to think about though. I think mriedem is right that it would skip an already migrated one even if it got a stale view of the shared set | |
| 21:43:02 | melwitt | and set doesn't raise if you add the same element twice, just a no-op | |
| 21:44:06 | mriedem | looks like dtantsur|afk's change in ironic to test this is blowing up during scheduling http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | |
| 21:44:20 | mriedem | edleafe: i left a couple of comments/questions in https://review.openstack.org/#/c/487954/ | |
| 21:46:09 | mriedem | http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | |
| 21:46:09 | mriedem | hmm, the compute node should be reporting inventory to placement which would include the CUSTOM_BAREMETAL resource class | |
| 21:50:36 | mriedem | looks like we don't get inventory data before that | |
| 21:55:28 | mriedem | i'm a bit torn on if we should land the nova change w/o actually seeing this working in the devstack test patch for ironic | |
| 21:55:40 | mriedem | and i have to run right now anyway to pick up maya | |
| 21:56:25 | melwitt | should I remove my vote for now? | |
| 21:56:30 | mriedem | melwitt: no it's ok | |
| 21:56:40 | mriedem | i'm on the fence about just sending it in and then fixing later if there is something wrong | |
| 21:56:49 | melwitt | k | |
| 21:56:55 | mriedem | would be nice if someone from ironic could say, i tested this manually and it's fine | |
| 21:57:00 | mriedem | edleafe: ^ did anyone test manually? | |
| 21:57:12 | edleafe | mriedem: not that I know of | |
| 21:58:38 | mriedem | ok i'll ask in -ironic but i think most of that team is gone for the day | |
| 21:58:42 | mriedem | bbibab | |
| 21:58:44 | mriedem | *bbiab even | |
| 22:01:02 | openstackgerrit | Merged openstack/nova master: Improve stable-api doc with current API state https://review.openstack.org/489926 | |
| 22:18:09 | bauzas | folks, see you tomorrow | |
| 22:18:27 | bauzas | will look at the branch if any | |
| 22:27:37 | edleafe | So it looks like when Ironic sets the resource_class for a node, it doesn't do anything to create that in Placement | |
| 22:28:00 | edleafe | which would explain http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | |