| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 14:52:52 | dansmith | tssurya: no, not really, that's just because we have to wait for all the results in order to be able to produce them in order | |
| 14:53:02 | tssurya | mriedem: yea its still going to 70 cells out of which we care only about the 1 cells which might have those 10 nodes | |
| 14:53:19 | mriedem | hmm, wonder if we could front-filter the cells via host mappings, | |
| 14:53:24 | mriedem | that wouldn't be worth it in the case of 1-2 cells, | |
| 14:53:29 | mriedem | but for 70 it might | |
| 14:53:50 | dansmith | mriedem: like we do for projects on list.., and we have a knob to choose | |
| 14:53:55 | mriedem | sort of like how we get instance mappings by project_id and then filter the cells from those mappings | |
| 14:53:57 | mriedem | yeah | |
| 14:54:31 | tssurya | mriedem: yea we had a bug for that, and kind of have a patch to try that out, will first see if there is a major difference | |
| 14:55:03 | mriedem | diliprenkila: i don't see os_hidden here https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | |
| 14:55:07 | mriedem | is it just not documented? | |
| 14:55:13 | dansmith | mriedem: and, it only helps for 70 cells when you regularly cut out 68 of those cells for any given scheduling request | |
| 14:55:31 | tssurya | dansmith: oh ok | |
| 14:56:02 | bauzas | mriedem: diliprenkila: I guess he means kvm_hidden | |
| 14:56:07 | openstack | Launchpad bug 1767303 in OpenStack Compute (nova) "Scheduler connects to all cells DBs to gather compute nodes info" [Undecided,Incomplete] - Assigned to Surya Seetharaman (tssurya) | |
| 14:56:07 | mriedem | tssurya: this bug? https://bugs.launchpad.net/nova/+bug/1767303 | |
| 14:56:16 | tssurya | yea | |
| 14:56:35 | mriedem | bauzas: no it's a property on the image? https://github.com/openstack/glance/blob/a308c444065307e99f18b521ed8d95714be24da7/glance/db/sqlalchemy/alembic_migrations/versions/rocky_expand01_add_os_hidden.py#L16 | |
| 14:57:12 | bauzas | ah my bad then | |
| 14:57:34 | bauzas | what does this ? | |
| 14:58:15 | bauzas | mmm https://github.com/openstack/nova/search?q=os_hidden&unscoped_q=os_hidden | |
| 15:00:10 | mriedem | tssurya: ok changed that to triaged and added notes | |
| 15:00:24 | mriedem | added to https://etherpad.openstack.org/p/BER-cells-v2-updates also | |
| 15:00:31 | mriedem | since i've got a running list of perf related issues in there | |
| 15:01:29 | sean-k-mooney | bauzas: looking at http://codesearch.openstack.org/?q=os_hidden&i=nope&files=&repos= os_hidden is used by horizon and in glance but not in nova | |
| 15:02:36 | mriedem | nova likely gets the image properties, shoves them into instance.system_metadata, and then on snapshot we populate the image meta for the new image from that sys_meta but aren't saying os_hidden is a boolean | |
| 15:02:41 | mriedem | and default to send it as a string or something | |
| 15:03:00 | mriedem | we have a whitelist for shit like this | |
| 15:03:03 | tssurya | mriedem: thanks | |
| 15:04:21 | diliprenkila | mriedem: can't we send os_hidden as boolean? | |
| 15:05:41 | sean-k-mooney | diliprenkila: we proably could but nova does not know os_hidden is a thing so it has no logic specifically for haneling it | |
| 15:05:42 | mriedem | diliprenkila: the problem is likely here https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/image/glance.py#L734 | |
| 15:05:56 | mriedem | this is why microversions are nice - nova, as a client, never asked for this new response field | |
| 15:06:03 | mriedem | stashed it away and then blindly gave it back | |
| 15:06:41 | mriedem | apparently there isn't any tempest testing for this either | |
| 15:08:18 | mriedem | diliprenkila: i left some questions in the bug, i'm not sure why tempest wouldn't already be failing on this, unless you have to do something to trigger this failure - can you provide reproduction steps in the bug report? | |
| 15:08:54 | diliprenkila | mriedem : yes i will provide reproduction steps | |
| 15:09:33 | mriedem | i'm pretty sure "output[prop_name] = str(prop_value)" is the problem | |
| 15:09:55 | sean-k-mooney | is this something the glanceclient could fix for us | |
| 15:10:06 | sean-k-mooney | it defines it as a bool in its schema | |
| 15:10:08 | sean-k-mooney | http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/v2/image_schema.py#n216 | |
| 15:10:08 | mriedem | we don't use glanceclient | |
| 15:10:15 | sean-k-mooney | oh ok | |
| 15:10:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421 | |
| 15:10:18 | mriedem | i think we only use ksa now | |
| 15:10:48 | sean-k-mooney | we still import glancclient | |
| 15:11:27 | mriedem | oh yeah i guess we do, | |
| 15:11:35 | mriedem | we use ksa to get the session adapter thing to construct glanceclient | |
| 15:12:04 | mriedem | anyway, it'd be cool if apis didn't return things you didn't ask for... | |
| 15:12:37 | diliprenkila | mriedem: yes | |
| 15:12:41 | sean-k-mooney | so could we modify glance client to munge the values form strings to bools so we did not have to handel this field ourselves | |
| 15:15:30 | mriedem | i'm sure if we complained to the glance team about this, they'd say "you should be using the schema provided with the image" | |
| 15:15:35 | mriedem | which we aren't | |
| 15:15:55 | mriedem | there is only one image field we deal with the schema and that's disk_format | |
| 15:16:20 | mriedem | https://review.openstack.org/#/c/375875/ | |
| 15:19:53 | diliprenkila | mriedem: so we should fix the os_hidden type in nova ? not in glance | |
| 15:20:56 | openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195 | |
| 15:21:59 | mriedem | diliprenkila: i don't think there is probably anything to change in glance, | |
| 15:22:14 | mriedem | but i'd like to know why tempest isn't failing with this, but i need to know the recreate steps, | |
| 15:22:15 | openstackgerrit | Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 15:22:30 | mriedem | because tempest has very basic tests where it creates a server and then creates a snapshot of that server, | |
| 15:22:40 | mriedem | which i would think should cause this failure | |
| 15:26:27 | mriedem | maybe tempest isn't using image api v2.7/ | |
| 15:26:29 | mriedem | ? | |
| 15:27:59 | diliprenkila | mridem: may be | |
| 15:28:06 | artom | mriedem, I think by default it goes to the lowest microversion | |
| 15:28:21 | mriedem | glance doesn't have microversions but... | |
| 15:28:22 | artom | Unless the test specifies min_microversion (or just microversion?) | |
| 15:28:33 | artom | Oh, so just endpoints? | |
| 15:28:56 | mriedem | i need a mordred | |
| 15:29:14 | mordred | I didn't do it | |
| 15:29:19 | mriedem | image api versions, go! | |
| 15:29:23 | mriedem | as in, wtf | |
| 15:29:25 | artom | You're holding out for a mordred 'till the end of the night | |
| 15:29:30 | mriedem | does the user opt into those or you just get what the server has available? | |
| 15:29:31 | diliprenkila | mriedem: i am using nova: 18.0.0 , glance: 2.9.1 | |
| 15:29:54 | mordred | they're silly. there is no selection - you just get the API described by the highest number in that list | |
| 15:30:06 | mordred | so - basically - ignore the thing after the 2. | |
| 15:30:17 | mriedem | so if glance is rocky, i get 2.7 | |
| 15:30:22 | mriedem | https://developer.openstack.org/api-ref/image/versions/index.html#version-history | |
| 15:30:26 | mordred | yeah | |
| 15:30:40 | openstack | Launchpad bug 1806239 in OpenStack Compute (nova) "nova-api should handle type conversion while creating server snapshots " [Undecided,New] | |
| 15:30:40 | mriedem | then i don't know why tempest would not fail on https://bugs.launchpad.net/nova/+bug/1806239 | |
| 15:31:07 | mordred | nova is still using glanceclient right? | |
| 15:31:12 | mriedem | yeah | |
| 15:31:33 | mordred | yeah - tempest uses direct rest calls - so it's possible glanceclient is doing something wrong. or tempest is doing something wrong | |
| 15:32:29 | mriedem | well, tempest would just create a server and tell nova to snapshot it | |
| 15:32:33 | mriedem | and then nova will use glanceclient | |
| 15:32:52 | mriedem | if that's all it takes to tickle this with rocky glance, i'm not sure why tempest wouldn't blow up | |
| 15:33:37 | mriedem | anyway, i'll wait for diliprenkila to provide recreate steps | |
| 15:33:59 | mordred | oh. gotcha | |
| 15:34:05 | mordred | yeah. that's super weird | |
| 15:35:10 | ShilpaSD | Hi All, facing issue while stack, E: Sub-process /usr/bin/dpkg returned an error code (1), any suggestions to resolve this? | |
| 15:37:28 | dansmith | mriedem: so on that rpc logging thing at startup, do we think the actual query is slowing things down, or the logging of the pointless message? | |
| 15:37:47 | mriedem | idk | |
| 15:38:26 | mriedem | looking at logs | |
| 15:39:20 | mriedem | well we take 2 seconds dumping our gd config options :) | |
| 15:39:35 | dansmith | which should be mostly just log traffic, right? | |
| 15:39:40 | mriedem | yeah | |
| 15:39:41 | dansmith | so maybe the logging is the thing | |