| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-15 | |||
| 18:37:47 | sean-k-mooney | o/ | |
| 18:58:33 | _erlon_ | mriedem: only 1 | |
| 18:58:40 | _erlon_ | its a devstack node | |
| 19:00:02 | mriedem | _erlon_: oh, well, make sure it's reporting into placement with the correct inventory | |
| 19:00:15 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-list | |
| 19:00:20 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list | |
| 19:00:34 | _erlon_ | mriedem: its a devstack node and should be working fine | |
| 19:00:46 | _erlon_ | I can like create up to 3 instances | |
| 19:00:59 | _erlon_ | the 4th fails for some reason | |
| 19:02:25 | mriedem | well, you can use ^ to see the inventory of the provider, | |
| 19:02:32 | mriedem | and https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-usage-show to see the usage of the project which you used to create instances | |
| 19:02:42 | mriedem | and then see if there is any remaining capacity for the 4th instance you're trying to create | |
| 19:34:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: conf: Undeprecate and move the 'dhcp_domain' option https://review.openstack.org/480616 | |
| 19:34:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle unsetting '[DEFAULT] dhcp_domain' https://review.openstack.org/652662 | |
| 19:49:32 | mriedem | edleafe: error response status code question! https://review.openstack.org/#/c/645458/17/specs/train/approved/add-host-and-hypervisor-hostname-flag-to-create-server.rst@141 | |
| 19:50:01 | mriedem | do those still exist? | |
| 19:50:08 | edleafe | jink | |
| 19:50:11 | edleafe | jinx | |
| 19:50:16 | aspiers | we still have some | |
| 19:50:21 | aspiers | but I don't think they tend to work | |
| 19:50:30 | aspiers | unless you're a timelord | |
| 19:50:37 | jaypipes | _erlon_: did you find the answer to your query? | |
| 19:52:25 | artom | Just quick sanity check to be 100% sure, but the resource stats we provide in https://developer.openstack.org/api-ref/compute/?expanded=show-hypervisor-details-detail#show-hypervisor-details come from the resource tracker via the ComputeNode object, right? And specifically, not from placement? | |
| 19:52:31 | edleafe | mriedem: that's not really a response code question, is it? If the app logic is to let the scheduler pick, then 2xx. If you treat the value as missing, then 400 | |
| 19:53:02 | mriedem | in the same request, the API will return a 409 HTTP Conflict error." | |
| 19:53:02 | mriedem | (``az:host:node``) mechanisms are mutually exclusive. If both are specified | |
| 19:53:02 | mriedem | edleafe: this one? "The new (``host`` and/or ``hypervisor_hostname``) and the old | |
| 19:53:11 | mriedem | they are saying the api will return an error and we don't get to the scheduler | |
| 19:53:16 | mriedem | i'm just saying that's not a 409 | |
| 19:53:49 | edleafe | mriedem: huh, I clicked that link and L125 was front and center | |
| 19:54:01 | mriedem | artom: everything in the hypervisors api comes from the compute node yeah | |
| 19:54:08 | artom | mriedem, cheerS! | |
| 19:54:27 | edleafe | Yeah, that's not a 409. There's nothing conflicting about the state of the system and the request. It's just a bad request | |
| 19:55:41 | mriedem | artom: interestingly that API doesn't use the ComputeNode.stats object https://github.com/openstack/nova/blob/78e742662edd164c46382c31e106884762fed029/nova/db/sqlalchemy/api.py#L730 | |
| 19:55:58 | mriedem | https://github.com/openstack/nova/blob/78e742662edd164c46382c31e106884762fed029/nova/objects/compute_node.py#L81 | |
| 19:56:46 | mriedem | which is this happy thing https://github.com/openstack/nova/blob/78e742662edd164c46382c31e106884762fed029/nova/compute/stats.py | |
| 19:56:52 | artom | mriedem, should it? Looks like stats is an aggregate, while the hypervisors API is per node | |
| 19:57:41 | mriedem | i'd say my default answer to "should the hypervisors api have more detail in it" is "probably not' | |
| 19:58:22 | mriedem | https://developer.openstack.org/api-ref/compute/?expanded=#show-hypervisor-statistics is an aggregate yes | |
| 19:58:30 | artom | I meant "should it" as more of a "why would you expect it to?" | |
| 19:59:58 | artom | Oooo I know, let's have NUMA nodes as REST resources and provide an API to access /sys/devices/system/node/node0/meminfo | |
| 20:00:23 | mriedem | from the outside i'd expect an api that is called "statistics" to leverage a 'stats' field on that resource in the db | |
| 20:00:49 | mriedem | artom: i'm pretty sure there is an intel spec for that | |
| 20:00:59 | artom | *snerk* | |
| 20:03:17 | artom | "efried pto" Aha, that explains it ;) | |
| 20:04:18 | mriedem | mnaser: i saw you abandoned https://review.openstack.org/#/c/577933/ - is that because you don't need it anymore, or just don't have time to work on it? because i think a couple of people asked for the same thing and it's reasonable imo, probably just needs paperwork done and such (which someone could help with) | |
| 20:04:56 | mnaser | mriedem: I still think it's useful but I don't think I have the time to work on the paperwork and what not honestly | |
| 20:05:05 | mnaser | not right now, anyways | |
| 20:05:25 | mnaser | maybe I can restore and leave it with a "hey if you wanna pick this up, feel free to?" | |
| 20:06:10 | mriedem | i think the only question is about a stale hostId value in the config drive on an instance that is live migrated around | |
| 20:07:04 | mriedem | but for that matter the az value in the config drive could already be stale for the same reason | |
| 20:07:08 | artom | mriedem, didn't we specifically make a point of not updating device tag information on the config drive? | |
| 20:07:23 | artom | hostId could fall under the same argument - "if you need up to date info, use the API" | |
| 20:07:45 | mriedem | artom: i vaguely recall talking about punting on updating tags in the config drive because we just don't refresh the thing | |
| 20:08:02 | mriedem | we rebuild it in some cases (rebuild w/o shared storage, unshelve) | |
| 20:08:33 | artom | I believe we settled on something like "we won't update it for you, it'll get rebuilt when it gets rebuilt" | |
| 20:08:48 | mriedem | you get what you get and you don't throw a fit? | |
| 20:11:00 | mriedem | mnaser: i'll reach out to our downstream product team and see if they have a similar need for this | |
| 20:11:06 | artom | mriedem, basically - https://github.com/openstack/nova/blob/master/releasenotes/notes/virt-device-tagged-attach-53e214d3b3fdd183.yaml | |
| 20:11:38 | mnaser | mriedem: cool, want me to restore it? | |
| 20:11:52 | mriedem | mnaser: i can if i'm going to work on it | |
| 20:11:56 | mriedem | so don't bother | |
| 20:12:05 | mnaser | mriedem: ok cool! | |
| 20:12:22 | mnaser | I have been dealing with other fun weird things | |
| 20:12:33 | mnaser | like resource reporting messages taking way too long | |
| 20:12:36 | mnaser | and timing out | |
| 20:12:43 | mnaser | but that only happening during live migrations | |
| 20:13:01 | mnaser | I don't wanna point at nova yet.. but it's pretty weird. | |
| 20:13:17 | mriedem | "resource reporting messages"? | |
| 20:13:33 | mnaser | eh the stuff that runs every little bit and reports the instance list to the scheduler | |
| 20:13:45 | mnaser | and then at some point a post_live_migration failed too with a timeout | |
| 20:13:47 | mriedem | oh, didn't know you relied on that | |
| 20:14:03 | mnaser | I mean I think if we want (anti-)affinity, we have to afaik? | |
| 20:14:03 | mriedem | sync_instance_info should be a broadcast - no timeout | |
| 20:14:18 | mriedem | no, it's just a cache in the scheduler | |
| 20:14:35 | mriedem | if the data isn't cached in the scheduler it pulls it from the db per request - which could be slow | |
| 20:14:39 | mnaser | oh yeah I remember that | |
| 20:14:47 | mnaser | let me find the traceback | |
| 20:15:00 | dansmith | mriedem: I have to submit something to lower-constraints itself before I can put it in nova yeah? | |
| 20:15:13 | mriedem | dansmith: no, it's per-project | |
| 20:15:18 | mriedem | not like the old g-r way | |
| 20:15:24 | dansmith | sigh | |
| 20:15:30 | mriedem | honestly i don't know why we have lower-constraints.txt and requirements.txt | |
| 20:15:32 | dansmith | might as well not ever try to remember because it'll be different | |
| 20:15:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image type capability flags and trait conversions https://review.openstack.org/652710 | |
| 20:15:51 | mriedem | g-r was dropped so requirements.txt isn't automatically updated from g-r anymore | |
| 20:16:01 | mnaser | https://www.irccloud.com/pastebin/drWAr4hV/ | |
| 20:16:04 | mriedem | and lower-constraints was added, but.... | |
| 20:16:07 | dansmith | right but I thought lower was, but whaterver | |
| 20:16:16 | openstackgerrit | Merged openstack/nova master: Remove cells v1 jobs https://review.openstack.org/651289 | |
| 20:16:17 | mnaser | is resource_tracker not thee same thing? maybe I'm being misleading here | |
| 20:16:36 | mriedem | mnaser: not that's something else, sec | |
| 20:17:10 | mriedem | https://github.com/openstack/nova/blob/stable/stein/nova/compute/resource_tracker.py#L1323 | |
| 20:17:19 | mriedem | it's timing out pulling instances from the db | |
| 20:17:34 | mriedem | well, "an instance" | |
| 20:17:50 | mriedem | which....your db / mq must be getting hammered? | |
| 20:18:24 | mnaser | mriedem: nothing sticks out | |
| 20:18:39 | mriedem | that will kill the whole update_available_resource periodic though | |
| 20:18:50 | mriedem | until the next time it runs i mean | |
| 20:19:35 | mriedem | sync_instance_info is a different periodic that reports info to the scheduler | |