| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 23:24:04 | eandersson | http://paste.openstack.org/show/754875/ | |
| 23:24:15 | eandersson | sean-k-mooney, I honestly don't think it's a rmq issue directly | |
| 23:25:00 | sean-k-mooney | ok its unlikely that its related to that placement error | |
| 23:25:17 | sean-k-mooney | that should happen before we try to build the instace | |
| 23:25:29 | eandersson | That log was from the same milisecond | |
| 23:25:58 | eandersson | I found two Messaging errors and both are the same issue | |
| 23:26:00 | sean-k-mooney | its posible that we only have 1 candiate host | |
| 23:26:06 | sean-k-mooney | and that we raced | |
| 23:26:10 | eandersson | Other than that no other oslo messaging issues | |
| 23:26:27 | sean-k-mooney | as a result of the fact overcommit is not working | |
| 23:26:41 | eandersson | If there was a rmq issue I am sure other services or request would have failed | |
| 23:26:56 | sean-k-mooney | neutron would be the first | |
| 23:26:57 | eandersson | We have a pretty massive deployment so always a lot of things going on | |
| 23:27:18 | sean-k-mooney | althogh they have redused the rpc traffic a lot in the last releae or two | |
| 23:29:59 | sean-k-mooney | oh the same message is also logged form here https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L676 | |
| 23:30:04 | artom | Heh, for what it's worth, that per-compute libvirt connection mocking has issues: | |
| 23:30:08 | artom | " 2019-07-25 19:29:16,889 ERROR [nova.virt.libvirt.host] Hostname has changed from test_compute0 to test_compute1. A restart is required to take effect." | |
| 23:30:10 | sean-k-mooney | the other place was teh cellv1 version | |
| 23:31:18 | sean-k-mooney | actully never bind its the same function | |
| 23:33:17 | sean-k-mooney | so if this is a retry then we are executing https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L642-L676 | |
| 23:34:08 | sean-k-mooney | and on line 655 we are trying to calim the alternate hosts which is failing with http://paste.openstack.org/show/754875/ because overcommit is not working | |
| 23:34:54 | eandersson | Yea pretty sure the error is within oslo.messaging | |
| 23:35:16 | sean-k-mooney | it might not be | |
| 23:35:25 | sean-k-mooney | when we rais the excetip form here https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L676 | |
| 23:35:33 | sean-k-mooney | it is not caut locally in this funciton | |
| 23:35:33 | eandersson | oh | |
| 23:39:17 | sean-k-mooney | i should proably go to bed but there is likely a bug there but im to tired to track it this evening | |
| 23:40:04 | mnaser | efried: i think you were working on moving nova to use openstacksdk ? do you have some of the commit you did in nova that did that? trying to do something similar for another project | |
| 23:42:19 | eandersson | Thanks for the help sean-k-mooney | |
| 23:42:24 | eandersson | I'll create a bug report | |
| 23:42:46 | gmann | johnthetubaguy: melwitt sean-k-mooney efried added my comment/query on unified limit spec. I am not very clear about how we will handle GET for limits which are not going to move to new unified limits(for example server_groups). | |
| 23:43:29 | gmann | I am almost ok for proxy APIs (as per operators interest) if HTTPGone on those APIs is not acceptable. | |
| 23:45:35 | melwitt | gmann: thanks for reviewing. I am not 100% operators would be opposed to having to use an older microversion to use the APIs *but* I think the thing that makes it weird is that unified limits would be opt-in. so that is where I'm uneasy with removing proxy API in new microversion. wanted your opinion on that aspect as well | |
| 23:50:17 | gmann | melwitt: 'removing proxy API in new microversion' and keep them working for older one seems no benefit and even more maintenance. I was thinking we say those APIs are gone (410 response code HTTPGone) because nova quota system is gone and without microversion bump. similar approach as nova-network & n-cert case. | |
| 23:51:16 | gmann | and before we do that we can trigger the notification to users via deprecating those APIs | |
| 23:51:29 | melwitt | gmann: oh, sorry, I didn't know HTTPGone is different. my bad | |
| 23:52:07 | melwitt | yeah, I'd like to be able to do that but wasn't sure about the API perspective of deprecating something when the new thing is opt-in and not on by default | |
| 23:52:41 | gmann | melwitt: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/floating_ip_dns.py#L25 | |
| 23:53:02 | openstack | Launchpad bug 1837955 in OpenStack Compute (nova) "MaxRetriesExceeded sometime fails with messaging exception" [Undecided,New] | |
| 23:53:02 | eandersson | sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1837955 | |
| 23:54:42 | gmann | melwitt: the only things make me feel uncomfortable to do that was the comment johnthetubaguy added in alternate section about Forum discussion with operators about keeping the old tooling. But i hope that is only for transition period not permanently | |
| 23:57:27 | gmann | melwitt: and second point is about GET quotas API to get the limits which are not going on new system. keep existing GET quotas APIs for them ? | |
| 23:57:52 | melwitt | limits mode is on by default | |
| 23:57:52 | melwitt | gmann: ok. the treatment of /os-quota-sets and /os-quota-class-sets is definitely temporary for the transition. the /limits API we're not completely sure because if we HTTPGone that one, there will be no more ability for users to show limits + usage in one API. but if community could be OK with having to go to two different API (keystone for limits and placement for usage) then we could 410 /limits too in the future when unified | |
| 23:59:10 | gmann | melwitt: +1 we can decide the limit thing later once hierarchy unified limits are there. | |
| #openstack-nova - 2019-07-26 | |||
| 00:01:06 | gmann | melwitt: but can we query on placement for nova owned limits usage (server groups, injected_files etc) ? also not about showing the limit from keystone so we need to keep the GET /os-quota-sets for those limits al least. | |
| 00:01:45 | melwitt | gmann: I'm not sure I understand the second question about the existing GET quotas API for limits. we have to keep that as long as old quotas are the default, I think | |
| 00:02:11 | melwitt | gmann: we cannot, no. so that's another thing | |
| 00:03:17 | gmann | yeah. | |
| 00:04:35 | melwitt | gmann: it's a little unclear because there's overlap with /limits and GET /os-quota-sets. I'm not sure we need the latter if we have the former (once unified limits is default down the road) | |
| 00:05:17 | melwitt | the (server groups, injected_files etc) limits are shown from /limits and will only be modifiable via the config files. no more API access for those | |
| 00:05:35 | melwitt | *no more API access to update those | |
| 00:06:13 | gmann | yeah, if we keep /limits then it is enough. But we need one API to show those limits and usage. | |
| 00:06:41 | melwitt | yeah, agreed | |
| 00:12:18 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 00:20:43 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 00:22:03 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 03:00:44 | openstackgerrit | karim proposed openstack/nova-specs master: Support VM memory encryption using Intel's MKTME https://review.opendev.org/666769 | |
| 03:22:36 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Always cleanup when hitting MaxRetriesExceeded https://review.opendev.org/672855 | |
| 03:23:42 | openstackgerrit | pengyuesheng proposed openstack/os-vif master: Blacklist sphinx 2.1.0 (autodoc bug) https://review.opendev.org/672856 | |
| 03:25:24 | eandersson | sean-k-mooney, no clue if that is the fix, but looks reasonable to me ^ | |
| 03:26:08 | openstackgerrit | pengyuesheng proposed openstack/os-vif master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/672857 | |
| 03:27:17 | openstackgerrit | pengyuesheng proposed openstack/os-vif master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/672857 | |
| 03:46:35 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Always cleanup when hitting MaxRetriesExceeded https://review.opendev.org/672855 | |
| 03:49:14 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Always cleanup when hitting MaxRetriesExceeded https://review.opendev.org/672855 | |
| 04:25:41 | openstackgerrit | Merged openstack/nova-specs master: Add user_id to the migrations https://review.opendev.org/667894 | |
| 04:31:37 | alex_xu | sean-k-mooney: stephenfin the real_time_mask indeed neeed the '^', you can check my reply https://review.opendev.org/#/c/668656/4/specs/train/approved/use-pcpu-vcpu-in-one-instance.rst@84 | |
| 04:49:20 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 07:02:41 | openstackgerrit | Huachang Wang proposed openstack/os-traits master: CPU: add a trait for AVX512-VNNI support https://review.opendev.org/672888 | |
| 07:48:57 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 07:48:57 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 07:48:58 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 07:48:58 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 07:48:59 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 07:48:59 | openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 07:51:36 | openstackgerrit | Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782 | |
| 07:51:37 | openstackgerrit | Alex Xu proposed openstack/nova master: Calling the virt driver's claim/unclaim_for_instance in resource tracker https://review.opendev.org/670784 | |
| 07:51:37 | openstackgerrit | Alex Xu proposed openstack/nova master: Moves the allocation retrieving early https://review.opendev.org/670783 | |
| 07:51:38 | openstackgerrit | Alex Xu proposed openstack/nova master: Populates the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 07:51:38 | openstackgerrit | Alex Xu proposed openstack/nova master: Add DeviceManager to the libvirt virt driver https://review.opendev.org/671388 | |
| 07:51:39 | openstackgerrit | Alex Xu proposed openstack/nova master: Adds functional test for creating the instance with vgpus https://review.opendev.org/671398 | |
| 07:51:39 | openstackgerrit | Alex Xu proposed openstack/nova master: Using the claim/unclaim_for_instance for mdevs https://review.opendev.org/671222 | |
| 10:11:55 | aspiers | sean-k-mooney, efried, bbobrov: exciting progress! http://paste.openstack.org/show/754889/ | |
| 10:13:21 | aspiers | cdent: inventory stuff seems to be working nicely :) ^^^ | |
| 10:14:01 | cdent | excellent, glad to hear it | |
| 10:42:27 | sean-k-mooney | aspiers: yep that does look like its working well | |
| 10:42:59 | sean-k-mooney | aspiers: you said you finally booted an sev instance with openstack yesterday right or is that the goal for today | |
| 10:43:35 | sean-k-mooney | aspiers: cdent actully should th max unit on that inventory not be 1 | |
| 10:44:02 | sean-k-mooney | you can only have one mem_encryption_context per vm right | |
| 10:45:19 | openstackgerrit | sean mooney proposed openstack/os-vif master: only disable mac ageing for ovs hybrid plug https://review.opendev.org/672834 | |
| 10:45:33 | sean-k-mooney | this a relitivly minor nit | |
| 10:55:12 | aspiers | sean-k-mooney: booting is the goal for today | |
| 10:56:34 | aspiers | sean-k-mooney: I don't get why you think max_unit should be 1? there are 16 slots configured on that machine | |
| 10:56:57 | aspiers | ohhh, are you saying max_unit represents the maximum per VM, not the maximum per host? | |
| 10:56:57 | sean-k-mooney | yes but max_unit is the maxium amount you can request in a singe request | |
| 10:57:03 | sean-k-mooney | yes | |
| 10:57:04 | aspiers | ok I didn't know that | |
| 10:57:10 | aspiers | in that case yeah it should be 1 | |
| 10:57:12 | aspiers | good catch | |
| 10:57:14 | sean-k-mooney | total is the total capasity | |