| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 20:58:59 | artom | sean-k-mooney, well, you're in luck, the top-most patched failed the gate :) | |
| 20:59:09 | artom | So any feedback you have it's now or never | |
| 20:59:52 | melwitt | sean-k-mooney: thanks | |
| 21:01:40 | sean-k-mooney | artom: https://github.com/openstack/nova/blame/421c52d9d341b07d850c21e0e702a008a8e1d3b7/nova/compute/resource_tracker.py#L493-L495 | |
| 21:02:14 | sean-k-mooney | artom: when that runs it possible we have not registered teh compute node yet | |
| 21:02:28 | sean-k-mooney | if i remeber correctly | |
| 21:03:38 | artom | sean-k-mooney, I think that's changed since then... | |
| 21:03:50 | openstackgerrit | Merged openstack/nova master: Differentiate between InstanceNotFound and ConstraintNotMet https://review.opendev.org/c/openstack/nova/+/775309 | |
| 21:04:29 | openstackgerrit | Merged openstack/nova master: Add functional test for bug 1837995 https://review.opendev.org/c/openstack/nova/+/775449 | |
| 21:04:31 | openstack | bug 1837995 in OpenStack Compute (nova) ""Unexpected API Error" when use "openstack usage show" command" [Undecided,In progress] https://launchpad.net/bugs/1837995 - Assigned to melanie witt (melwitt) | |
| 21:04:43 | artom | Now it's only called from _setup_pci_tracker, which has the compute_node object available to it | |
| 21:05:01 | artom | sean-k-mooney, anyways, I've -1'ed the top-most patch that hasn't merged yet | |
| 21:05:08 | sean-k-mooney | yes but i think that has a similar beahvior | |
| 21:05:13 | artom | Take your time to review it | |
| 21:05:17 | artom | We can revisit this tomorrow | |
| 21:05:30 | sean-k-mooney | i was tryign to find it but you updated it and its hard to fid the old version :) | |
| 21:06:04 | sean-k-mooney | actully i can just go in the history i gues | |
| 21:06:04 | artom | I *only* changed node_id to compute_node | |
| 21:06:07 | artom | Nothing else | |
| 21:08:10 | sean-k-mooney | this is the placves where its called that i was worred about | |
| 21:08:11 | sean-k-mooney | https://github.com/openstack/nova/blob/1273c5ee0b18974d9837e9221fc9270429d428bf/nova/compute/resource_tracker.py#L727-L761 | |
| 21:08:36 | sean-k-mooney | well i think those case its fine | |
| 21:08:55 | sean-k-mooney | assuming create has the sideffect of it having the id | |
| 21:10:25 | sean-k-mooney | artom: why did you pass the compute node object in | |
| 21:10:27 | sean-k-mooney | instead of the id | |
| 21:10:40 | artom | Don't need to look it up then | |
| 21:10:49 | artom | Saves a DB query - gibi suggested it | |
| 21:11:01 | sean-k-mooney | id does not | |
| 21:11:14 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/778747/2/nova/pci/manager.py#64 | |
| 21:11:22 | sean-k-mooney | we were passing in the compute node id before | |
| 21:11:38 | sean-k-mooney | now we dont save it and just extract the id | |
| 21:12:01 | artom | We don't save what? | |
| 21:12:17 | sean-k-mooney | the compute node object in this object | |
| 21:17:16 | artom | We don't need to | |
| 21:17:47 | sean-k-mooney | you split out the chage that refactored the interface form the cahgne that used the compute node object | |
| 21:17:54 | artom | We save the numa_topology in PciDeviceStats | |
| 21:17:55 | artom | https://review.opendev.org/c/openstack/nova/+/774149/12/nova/pci/stats.py | |
| 21:18:03 | sean-k-mooney | that should not have been done imo | |
| 21:18:17 | artom | Debate with stephenfin on that, his idea :) | |
| 21:18:30 | artom | But to not throw him under the bus too much, I agree with it | |
| 21:18:34 | artom | Makes the changes cleaner | |
| 21:18:43 | artom | One to always pass compute_node instead of the optional node_id=None | |
| 21:18:46 | sean-k-mooney | not really | |
| 21:18:57 | sean-k-mooney | it was not optional | |
| 21:19:03 | artom | And another to pull the numa_topology from that and pass it to PciDeviceStats | |
| 21:19:07 | sean-k-mooney | it was a key word argument | |
| 21:19:18 | sean-k-mooney | but the comptue node was required in all the production code | |
| 21:19:25 | sean-k-mooney | *id | |
| 21:19:29 | artom | How is https://review.opendev.org/c/openstack/nova/+/778747/2/nova/pci/manager.py#54 not optional? | |
| 21:19:36 | artom | It literally says node_id=None | |
| 21:19:50 | sean-k-mooney | right but all uses of it out side fo test always set it | |
| 21:20:04 | artom | Yeah, my point exactly | |
| 21:20:11 | artom | Codify that it's always expected | |
| 21:20:28 | artom | And because we'll need the full compute_node later on, replace node_id with the full object | |
| 21:20:39 | sean-k-mooney | sure but not in a patch seperate form the new usage or the full object | |
| 21:22:21 | sean-k-mooney | artom: sorry this just annoys me because you were chanign someint i did not think you were going to change. it conflicts with my changes and it break my mental model of how the pci trakcer works | |
| 21:23:34 | artom | sean-k-mooney, that changes aren't that dramatic... | |
| 21:24:47 | sean-k-mooney | the main one was that it never store state in the pci tracker object directly | |
| 21:25:41 | sean-k-mooney | the numa toplogy object will have to be keep consitent now | |
| 21:26:23 | sean-k-mooney | ok we do sotre state but differently | |
| 21:26:54 | artom | We're talking about the *host* numa_topology | |
| 21:26:59 | sean-k-mooney | yes | |
| 21:27:00 | artom | When is that ever going to change... | |
| 21:27:57 | sean-k-mooney | within th elife time of the agent i guess it not going to | |
| 21:28:24 | sean-k-mooney | i mean memoy and cpu hotplug are thigns and you can reconfigure hyperthreading on the fly or hugepagers for that matter | |
| 21:28:40 | sean-k-mooney | hugepage is actully the most likely ot change at runtime | |
| 21:28:52 | sean-k-mooney | but to have that picked up you need to restart libvirtd | |
| 21:29:51 | sean-k-mooney | actully se also store the currently pinne cpu in the host numa toplogy blob | |
| 21:30:22 | sean-k-mooney | so its update every time we boot new vms | |
| 21:36:28 | artom | sean-k-mooney, the PCI tracker never uses that information though | |
| 21:36:45 | artom | I could add a comment to warn future programmers | |
| 21:37:08 | artom | We just need the socket/node mapping, and that's effectively constant | |
| 21:40:11 | lyarwood | melwitt / sean-k-mooney ; so in my defence, it's late and I shouldn't be working but when I said ~5mins earlier what I actually wanted to say was ~500 seconds. http://paste.openstack.org/show/803406/ | |
| 21:41:00 | sean-k-mooney | lyarwood: ay right that about right with paralle | |
| 21:42:01 | sean-k-mooney | i think i missed where you said ti too ~500 though | |
| 21:44:00 | sean-k-mooney | sub ten mintues is doable if you have good netowrking, io and a fast cpu | |
| 21:46:00 | lyarwood | yup I'm just running a 4 vCPU, 16GB, 50GB RAW disk VM on my p1 gen2 with a 1Gbps connection | |
| 21:52:11 | melwitt | lyarwood: heh, sorry, it wasn't only you, I had seen other mentions of 5-6 min about it before and your mention made me think to ask what am I doing wrong to not get this result 😆 | |
| 21:52:49 | sean-k-mooney | melwitt: yep dansmith was around the 5-8 minute mark | |
| 21:53:06 | sean-k-mooney | that i think was on baremetal | |
| 21:53:25 | sean-k-mooney | rather then nested virt but its doable | |
| 21:54:15 | dansmith | talking about devstack time? | |
| 21:54:22 | sean-k-mooney | yep | |
| 21:55:27 | sean-k-mooney | i think you were geting about 430 ish second if im not mistaken | |
| 21:55:30 | dansmith | yeah, I can do about 5mins with a less-than-full devstack config | |
| 21:55:49 | dansmith | with OCaaS plus parallel I can get 3xxx yeah | |
| 21:56:00 | dansmith | er, 3xx | |
| 21:56:11 | melwitt | what are the main things you disable? | |
| 21:56:26 | dansmith | disable_service c-bak etcd3 c-api c-vol c-sch swift horizon dstat | |
| 21:56:34 | dansmith | tempest if I don't need it | |
| 21:56:34 | melwitt | thanks | |
| 21:56:51 | sean-k-mooney | swift i think is kind of slow to set up | |
| 21:57:18 | sean-k-mooney | dstat should not make much of a differnce horizon take a while to complie and compress the static pages | |
| 21:57:52 | dansmith | well, not all of the systemctl commands are super fast | |
| 21:58:17 | dansmith | sometimes depending on what is running, daemon-reload can take a couple seconds, and start if it waits for the first child, etc | |
| 22:00:19 | sean-k-mooney | ya i notice that more on unstack then anything esle | |
| 22:00:45 | sean-k-mooney | some service take a long time to stop randomly | |
| 22:00:46 | dansmith | yeah, that's another good reason though, | |
| 22:01:01 | dansmith | more shtuff to unstack makes the process slower when you're iterating | |
| 22:02:10 | sean-k-mooney | yep although i normlaly see how long i can go with just doing sudo systemctl restart devstack@n-* | |