| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-14 | |||
| 19:56:08 | qqmber | hi there | |
| 22:41:37 | openstackgerrit | Ghanshyam proposed openstack/nova master: Reuse code from oslo lib for JSON policy migration https://review.opendev.org/c/openstack/nova/+/763487 | |
| 22:46:46 | openstackgerrit | Ghanshyam proposed openstack/placement master: [goal] Deprecate the JSON formatted policy file https://review.opendev.org/c/openstack/placement/+/764394 | |
| 22:54:24 | dansmith | melwitt: replied | |
| 22:55:03 | melwitt | dansmith: danke | |
| 22:55:59 | melwitt | I was coincidentally looking at some different gate failures involving presumable stress on node resources and wanted to see if there are similarities | |
| 22:56:43 | dansmith | well, this was a pretty standard OOM once I realized that, | |
| 22:57:09 | dansmith | but it's unusual that the OOM is caused by the test runner, so I hadn't seen an OOM manifest itself as a killing of one worker before | |
| 22:58:52 | melwitt | I see now, in syslog of the example you linked. literal oom-killer messages. what I've been looking at today is different | |
| 23:00:06 | dansmith | this one blows up so fast that it doesn't even register in dstat, | |
| 23:00:15 | dansmith | which is what took me a bit to realize it was an oom | |
| 23:00:28 | dansmith | it goes from like 45M to 1.5g in a couple seconds locally | |
| 23:00:33 | melwitt | wow | |
| 23:01:09 | melwitt | how were you able to tell it's the test runner? something in the logs? | |
| 23:01:45 | dansmith | well, the oom was a python process, and one worker (which is python) just never reports its results, | |
| 23:02:21 | dansmith | then reproing locally (with plenty of memory) I see one test runner blip to 1.5g, before exiting normally, which is the same size as the victim from the killer | |
| 23:02:42 | dansmith | I don't oom because I have lots of memory locally, but I could see that balloon, and if it happens at the wrong time in the gate, it'll trigger and die | |
| 23:03:06 | melwitt | I see, thanks | |
| 23:05:30 | melwitt | nice find. I've been looking at a lot of gate failures lately that seem to be outside of openstack and trying to find hints about what could be causing them | |
| 23:21:49 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Change default num_retries for glance to 3 https://review.opendev.org/c/openstack/nova/+/764963 | |
| 23:24:43 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 23:49:38 | sean-k-mooney | melwitt: actully regarding ^ | |
| 23:50:00 | sean-k-mooney | i realised that while the upper-constatis allows it im not sure lower constartis on stien does | |
| 23:50:50 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/stein/lower-constraints.txt#L79 oslo.concurrency==3.26.0 | |
| 23:51:19 | sean-k-mooney | it need 3.29.0 | |
| 23:51:42 | sean-k-mooney | are we allowed to bump it like that in a backport? | |
| 23:54:02 | sean-k-mooney | for what its worth i think its a good thing to backport just dont know what stable policy allows | |
| 23:54:06 | melwitt | oh yeah, we aren't. so I guess I was right originally, that it couldn't be backported | |
| 23:54:34 | melwitt | sigh | |
| 23:55:15 | sean-k-mooney | well with the patch on gerrit if peopel want to use it it should be eaiser for them | |
| 23:56:13 | sean-k-mooney | to do the backport you would have to add the fair lock form oslo to noava | |
| 23:56:32 | sean-k-mooney | and just use that instead | |
| 23:57:04 | melwitt | yeah, I doubt it's worth going that far | |
| 23:57:51 | sean-k-mooney | ya | |
| #openstack-nova - 2021-01-15 | |||
| 03:29:21 | openstackgerrit | Brin Zhang proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/c/openstack/nova/+/729563 | |
| 08:04:06 | gibi | dansmith: thanks for finding out reasone of the multistore failure an proposing a fix. nice catch! | |
| 08:17:31 | lyarwood | morning | |
| 08:18:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754700 | |
| 08:18:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_BLOCKDEV https://review.opendev.org/c/openstack/nova/+/754701 | |
| 08:18:05 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_S390X_CPU_COMPARE https://review.opendev.org/c/openstack/nova/+/754704 | |
| 09:11:34 | openstackgerrit | wushiming proposed openstack/os-vif master: Drop lower-constraints job https://review.opendev.org/c/openstack/os-vif/+/770907 | |
| 09:17:59 | nightmare_unreal | Hello , is there a way to force a VM to boot from volume. For e.g. during VM creation i want to enforce that it should be from volume only | |
| 09:19:21 | kashyap | lyarwood: Morning; for the NEXT_ version bump, we also traditionally give a heads-up on the openstack-discuss list -- wanna do that too? (E.g. one of my past emails: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010907.html) | |
| 09:22:29 | openstackgerrit | wushiming proposed openstack/python-novaclient master: Drop lower-constraints job https://review.opendev.org/c/openstack/python-novaclient/+/770912 | |
| 09:24:43 | lyarwood | kashyap: yup can do | |
| 09:25:14 | kashyap | Thank ya. | |
| 09:35:56 | songwenping_ | sean-k-mooney: Hi, what do you means about the vgpu spec: https://review.opendev.org/c/openstack/nova-specs/+/750116/17#L97, should we use rpc.cast the cyborg-agent to create a new mdev? | |
| 09:37:28 | lyarwood | kashyap: sent | |
| 09:39:43 | kashyap | lyarwood: Just read the mail in the archives; looks fine, thx. | |
| 09:41:59 | lyarwood | cool thanks | |
| 10:20:21 | songwenping_ | gibi: hi, about your comment in line https://review.opendev.org/c/openstack/nova/+/622336/20/nova/api/openstack/compute/remote_consoles.py#186, `if svc.version < |
|
| 10:21:14 | brinzhang0 | gibi: cyborg shelve/unshelve updated alex_xu's concern, pls re-check again, thanks https://review.opendev.org/c/openstack/nova/+/729563 | |
| 10:21:30 | gibi | songwenping_: I would go with http 400 | |
| 10:21:38 | gibi | brinzhang0: ack | |
| 10:22:38 | songwenping_ | gibi: get | |
| 10:23:06 | brinzhang0 | gibi: yes, the same as songwenping_'s concern , IMO 400 is more suitable | |
| 10:24:21 | brinzhang0 | gibi: thanks for check the cyborg shelve/unshelve patch ^^ | |
| 11:29:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: api: Log os-resetState as an instance action https://review.opendev.org/c/openstack/nova/+/770926 | |
| 11:43:24 | gibi | bauzas: are your concerns answered in https://review.opendev.org/c/openstack/nova/+/729563 ? I'm holding the +W until ack it | |
| 12:36:00 | sean-k-mooney | songwenping__: i was suggesting that gets shoudl not create teh mdev ever. the cyborg agent should create the mdevs for all bound arqs on start up. ideally it would set the binding state to "provisioning" on start up also and we would have anohter state "unknown" that the api would report if the cyborg agent misses its heart beat | |
| 12:37:12 | sean-k-mooney | songwenping__: so on start up nova would see 1 of 3 states, bound if the cyborg agent started first and hadn completed bininding, unknown if the cyborg agent has not heartbeat to the cyborg conductor yet | |
| 12:37:36 | sean-k-mooney | or provisioning if the agent is in the process of creating the mdev | |
| 12:38:32 | sean-k-mooney | if cyborg was to send the same bining complete event when it change teh status form provisioning to bound as it does during normal arq binding | |
| 12:38:45 | sean-k-mooney | the process in nova awould be the same | |
| 12:39:34 | sean-k-mooney | e.g. set up event handeler, check if its already bound if not wait for binding complete event, if it si bound cancel the even waiter and proceed with boot | |
| 12:40:55 | sean-k-mooney | gibi: bauzas does that sound like a resonable approch to ye instead of have get arq sometimes do an rpc call to the cybrog agent to create mdevs | |
| 12:41:37 | bauzas | gibi: will look at https://review.opendev.org/c/openstack/nova/+/729563 | |
| 12:42:28 | gibi | bauzas: thanks | |
| 12:42:57 | bauzas | and for the vGPU support in Cyborg, I'll also look at the new revision today | |
| 12:43:41 | gibi | sean-k-mooney: do we have this event handling at nova compute startup for neutron ports too? | |
| 12:44:48 | sean-k-mooney | gibi: we do not need to rebinding them in the neutron case just replug them. | |
| 12:45:18 | sean-k-mooney | in the neutorn case os-vif is the thing that attaches the ports to the network backend in general | |
| 12:45:37 | sean-k-mooney | which si the equivalent of creating the mdev | |
| 12:45:45 | gibi | sean-k-mooney: I see | |
| 12:47:47 | sean-k-mooney | i just find it unsetaling that we would considre change a get form a simple db lookup into an rpc call | |
| 12:48:25 | gibi | sean-k-mooney: so at compute startup nova gets the binding state from cyborg, if it is unknow or provisioning then keep the guest power state off but set up an event waiter. If the arq state is "OK" then nova would start the guest during compute startup. | |
| 12:48:45 | sean-k-mooney | gibi: not quite | |
| 12:48:50 | gibi | correct me please | |
| 12:49:30 | sean-k-mooney | i was thinking we would always set up the waiter and early out if it was bound like we do for normal spwan | |
| 12:50:02 | gibi | can nova we loose an event during the compute reboot? | |
| 12:50:29 | gibi | if yes then that guest would be stuck waiting of the event that was sent by cyborg but lost in the comptue restart | |
| 12:51:03 | gibi | if the event is never lost then I'm OK to wait for the event | |
| 12:51:05 | sean-k-mooney | the event would hit the api and then be enqued to the compute node topic queue | |
| 12:51:15 | sean-k-mooney | so i dont think it would be lost | |
| 12:51:15 | gibi | sean-k-mooney: cool | |
| 12:51:20 | gibi | that seem OK | |
| 12:51:34 | gibi | hm | |
| 12:52:12 | gibi | so in this case the sending the event is triggered by cyborg agent restart, in any other case sending the event is triggered by a nova bind request | |
| 12:52:39 | sean-k-mooney | yes | |
| 12:53:09 | gibi | so there are extra cases to handle. 1) a single cyborg agent restart will send events and if the compute service was not restarted then these events needs to be consumed but ignored | |
| 12:53:14 | sean-k-mooney | we could just call bind if we wanted too and not require teh cyborg agent to auto create them but i think the auto create would be more efficent | |
| 12:54:00 | sean-k-mooney | gibi: we have unexpeted event handeling in nova already | |
| 12:54:05 | gibi | cool | |
| 12:54:19 | sean-k-mooney | if we dont have a waiter when we deque it we just log it and discard | |
| 12:54:28 | gibi | that seems OK too then | |
| 12:54:34 | sean-k-mooney | which si ok because we will check the state when we get to that part of the code | |
| 12:55:44 | sean-k-mooney | basically im just suggesting using the exact saem event system we use of inital sapwn after where we start the binidng in the conductor then wait for it with an early out in the compute | |
| 12:56:03 | gibi | OK, I don't have a #2 actually :) | |
| 12:56:06 | sean-k-mooney | but in this case the cyborg agent would start the bind on start up | |
| 12:56:25 | gibi | sean-k-mooney: so far what you suggest feels OK to me | |