| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 02:33:33 | guilhermesp | the deployment has 160k rows in nova.instances | |
| 02:34:06 | guilhermesp | we currently re-mapped 80K instnaces, but sometimes the command finishes but there are still rows to remap | |
| 02:34:39 | guilhermesp | the active instnaces of the deployment are currently given 404, which mean they doesn't reside on the newly created cell during the deployment | |
| 02:35:33 | guilhermesp | we are sometimes adding to the command --max-counts like 50k, 100k and trying to track the remapping watching the rows in nova_api.instance_mappings | |
| 02:35:51 | guilhermesp | does anyone knows what could be the trick in that scenario | |
| 02:35:57 | openstackgerrit | Merged openstack/nova master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659201 | |
| 02:36:01 | guilhermesp | we have a staging env that this didn't happen | |
| 02:36:50 | guilhermesp | but we used to have around 1.8k rows in nova.instances | |
| 02:37:17 | guilhermesp | maybe the amount of records in the other env could be a possible reason? | |
| 04:47:53 | bhagyashris | sean-k-mooney: Hi, | |
| 07:20:41 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/stein: Fix assert methods in unit tests https://review.opendev.org/657708 | |
| 07:21:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Fix assert methods in unit tests https://review.opendev.org/657709 | |
| 07:21:46 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Fix assert methods in unit tests https://review.opendev.org/657709 | |
| 07:24:06 | openstackgerrit | zhangboye proposed openstack/nova-specs master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659470 | |
| 07:26:05 | kashyap | weshay: Was asleep when you pinged. | |
| 07:37:44 | kashyap | weshay: I see that melwitt (thanks) has already investigated the issue and posted a patch to skip logging the trace: https://review.opendev.org/#/c/659374/ | |
| 08:05:17 | openstackgerrit | Gorka Eguileor proposed openstack/nova master: Fix doc requirements https://review.opendev.org/659484 | |
| 08:05:58 | openstackgerrit | Gorka Eguileor proposed openstack/nova master: Fix doc requirements https://review.opendev.org/659484 | |
| 08:07:33 | openstackgerrit | Gorka Eguileor proposed openstack/nova master: Use os-brick locking for volume attach and detach https://review.opendev.org/614190 | |
| 09:10:08 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362 | |
| 09:28:22 | bhagyashris | stephenfin: Hi, Just want to ask regarding the reserved_host_cpus as we are setting the reserved value of cpus as 0 for both the PCPU and VCPU then how the reserved value will be maintain I mean there should some cpu reserved for host process | |
| 09:32:01 | sean-k-mooney | bhagyashris: that is achive by not listing them in the cpu_shared_set or the cpu_dedicated_set | |
| 09:33:17 | sean-k-mooney | so on an 8 core system you can do cpu_shared_set=2,3 cpu_dedicated_set=4-7 and that leave cores 0 and 1 for the host | |
| 09:34:01 | sean-k-mooney | this is how you reserve cores for the host using vcpu_pin_set today | |
| 09:34:04 | bhagyashris | sean-k-mooney: that means operator should keep in mind that some cpus should left for host process while setting the cpu_dedicated_set and cpu_shared_set | |
| 09:34:14 | sean-k-mooney | yes | |
| 09:34:27 | bhagyashris | okay | |
| 09:34:51 | bhagyashris | sean-k-mooney: thank you for clearing my doubt. | |
| 09:35:24 | sean-k-mooney | althogh today reserved_host_cpus defaults to 0 and by defalt the vcpu_pin_set if unset enable all host cpus | |
| 09:35:39 | sean-k-mooney | so if they dont do that it wont be a change in behavior | |
| 09:38:18 | bhagyashris | sean-k-mooney: okay, but suppose if operator assign all the cpus to the dedicated set then what will happen in that case, host process won't have any cpu to process the host process. | |
| 09:38:56 | sean-k-mooney | the hosts will steal cpu time form the pinned instances | |
| 09:39:26 | sean-k-mooney | but again that is what happens today if you dont ommit cores for the host to use form the vcpu pin set | |
| 09:41:03 | bhagyashris | okay, So that case should be handle by code or by operator then | |
| 09:41:20 | sean-k-mooney | by the operator | |
| 09:41:30 | bhagyashris | okay | |
| 09:42:14 | bhagyashris | Thank you :) | |
| 10:08:59 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362 | |
| 10:25:05 | mdbooth | Any reason https://review.opendev.org/658845 shouldn't be backported to stable? | |
| 11:59:23 | rabel | hi. there seems to be a problem with api microversions. i have an openstack rocky (deployed with packstack), but my openstackclient (newest version, directly from git) tells me that the compute api version is 2.1 | |
| 12:02:40 | artom | rabel, it's an known osc behaviour/bug. While novaclient autonegotiates to the greatest common supported microversion, osc defaults to 2.1, and you have to manually specify which microversion you want | |
| 12:13:03 | rabel | artom thanks | |
| 12:56:35 | alex_xu | efried: I think the in_tree take care of all the cases for https://review.opendev.org/#/c/645520/ also | |
| 13:08:37 | efried | alex_xu: Okay. I'll take a closer look at that patch soon. | |
| 13:08:50 | alex_xu | efried: thx | |
| 13:16:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374 | |
| 13:23:53 | mriedem | dansmith: random idea, but with something like https://bugs.launchpad.net/nova/+bug/1829349 where the RT is constantly failing b/c of conflicted pinned CPUs on the same host, we could leverage the build failure stats thing and if the RT fails after x hits or whatever we try to take that host out of scheduling | |
| 13:23:54 | openstack | Launchpad bug 1829349 in OpenStack Compute (nova) "Resource Tracker failed to update usage in case numa topology conflict happen" [Undecided,New] | |
| 13:24:31 | mriedem | they did the naughty and enabled live migration for numa instances, 2 landed on the same host with conflicting pinned CPUs, and how the RT constantly fails but the scheduler continues to pick the host b/c it looks free since it's usage isn't getting reported | |
| 13:24:42 | mriedem | s/how/now/ | |
| 13:25:08 | dansmith | oye | |
| 13:25:23 | dansmith | isn't the build failure count going up in that case anyway? | |
| 13:25:42 | mriedem | maybe, i can ask in the bug report | |
| 13:26:01 | sean-k-mooney | mriedem: did we fix the typo so that that config option works yet | |
| 13:27:47 | mriedem | the qemu vs QEMU thing? | |
| 13:28:01 | mriedem | it wouldn't matter, | |
| 13:28:09 | mriedem | because in this case they enabled the workaround to allow live migrating instances with numa | |
| 13:28:10 | sean-k-mooney | ya we were checking for hypervior = kvm but its alway qemu | |
| 13:29:19 | sean-k-mooney | yes we fixed it https://review.opendev.org/#/c/635350/] | |
| 13:29:51 | sean-k-mooney | we never actully released with it broken so we shoudl be good | |
| 13:29:53 | mriedem | well, this person is on rocky | |
| 13:29:54 | mriedem | but yes https://review.opendev.org/#/q/I5127227a1e3d76dd413a820b048547ba578aff6b | |
| 13:31:38 | sean-k-mooney | mriedem: i would have hoped the numa toplogy filter would have prevented new vms from being scudled by the way if there was not space | |
| 13:31:50 | sean-k-mooney | the one that was mighrtated would not be fixed but ya | |
| 13:32:32 | aarents | Hi, mdbooth I updated https://review.opendev.org/#/c/659054/ if you want to have a look, maybe also mriedem ? | |
| 13:32:38 | sean-k-mooney | which reminding me i have a meeting then i need to updated docs for the sriov migration | |
| 13:34:10 | mdbooth | aarents: Thanks, still +1. | |
| 13:34:13 | adrianc | sean-k-mooney: Hi, id like to see if we can converge on the FUP for SR-IOV LM, would be great if you can take a quick look :) | |
| 13:41:31 | sean-k-mooney | adrianc: sure i think this was more or less fine so ill take alook at the latest version | |
| 13:44:43 | adrianc | sean-k-mooney, i did not upload a new PS as i wanted to reach an agreement on the propery thing, anyway... you will see | |
| 13:56:17 | efried | Nova meeting in 4 minutes in #openstack-meeting | |
| 14:02:06 | mriedem | tssurya: i'm going to remove the locked reason stuff from the runways etherpad since the nova and novaclient stuff is done https://etherpad.openstack.org/p/nova-runways-train | |
| 14:09:31 | efried | sean-k-mooney: Last week's nova meeting we got a request to review | |
| 14:09:31 | efried | https://review.opendev.org/#/c/643023/ | |
| 14:09:32 | efried | https://review.opendev.org/#/c/643024/ | |
| 14:09:32 | efried | which it looked like you were actively looking into. Would you be able to follow up please? | |
| 14:10:05 | sean-k-mooney | yes | |
| 14:10:16 | tssurya | mriedem: ack | |
| 14:10:33 | efried | thanks sean-k-mooney | |
| 14:18:53 | sean-k-mooney | adrianc: just replied on https://review.opendev.org/#/c/659101/2/nova/objects/instance_pci_requests.py@47 | |
| 14:19:43 | adrianc | thx sean-k-mooney, ill take a look shortly | |
| 14:20:22 | sean-k-mooney | im going to go review https://review.opendev.org/#/q/topic:bug/1809095+(status:open+OR+status:merged) before my next meeting | |
| 14:21:01 | sean-k-mooney | which is your fix for bug #1809095 | |
| 14:21:02 | openstack | bug 1809095 in OpenStack Compute (nova) "Wrong representor port was unplugged from OVS during cold migration" [Medium,In progress] https://launchpad.net/bugs/1809095 - Assigned to Adrian Chiris (adrian.chiris) | |
| 14:32:32 | mriedem | btw, i remember mtreinish trying to get glance-api moved to wsgi in pike and it got hung up on threading stuff with eventlet, hence https://review.opendev.org/#/c/549743/ | |
| 14:32:57 | dansmith | yup | |
| 14:34:52 | mriedem | oh and https://review.opendev.org/#/c/531498/ | |
| 14:35:04 | mriedem | ^ taskflow evenlet async stuff + wsgi + glance-api | |
| 14:36:21 | mtreinish | yep, but I couldn't get anyone to ever really review that or explain how to use the async glance APIs. And there is no dsvm coverage for any of it | |
| 14:37:55 | mriedem | mtreinish: for context we're going over similar in nova-api https://review.opendev.org/#/c/650172/ | |
| 14:41:14 | cdent | I reckon the root of the problems in both glance and nova is that we are trying to async workings in an environment (an HTTP API) where it doesn't make sense. The traditional python solution to this problem is to dump into an offboard async task queue (celery and the like). The scatter/gather thing is not quite in the same boat as the glance stuff (which is more amenable to the celery thing perhaps) | |
| 14:46:13 | mtreinish | cdent: yeah, that was the best way to fix it, to just do the async bits in another service | |
| 14:46:37 | mtreinish | but I assumed just launching another process or thread to handle it should work as a quick workaround | |
| 14:46:56 | cdent | it ought to | |
| 14:47:24 | cdent | but I would guess that zombie blocked processes would present the same concern as blocked threads | |
| 14:47:36 | cdent | which might suggest: that's the way unix goes, deal... | |
| 14:53:11 | sean-k-mooney | i mean we could just not set a limit on the worker trhead in the tread pool or make it a config option | |
| 14:53:42 | sean-k-mooney | python will grow and srink the thread pool automatically if you dont specify the number of workers | |
| 14:53:53 | dansmith | that's the problem, these are a threadpool, which means they're re-used and capped at some max number, so enough api requests come in during a cell down event, and you immediately exhaust the pool of workers until things time out | |
| 14:54:15 | dansmith | sean-k-mooney: only up to N*CPUs I think I read | |