Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-06
15:05:42 mriedem diliprenkila: the problem is likely here https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/image/glance.py#L734
15:05:56 mriedem this is why microversions are nice - nova, as a client, never asked for this new response field
15:06:03 mriedem stashed it away and then blindly gave it back
15:06:41 mriedem apparently there isn't any tempest testing for this either
15:08:18 mriedem diliprenkila: i left some questions in the bug, i'm not sure why tempest wouldn't already be failing on this, unless you have to do something to trigger this failure - can you provide reproduction steps in the bug report?
15:08:54 diliprenkila mriedem : yes i will provide reproduction steps
15:09:33 mriedem i'm pretty sure "output[prop_name] = str(prop_value)" is the problem
15:09:55 sean-k-mooney is this something the glanceclient could fix for us
15:10:06 sean-k-mooney it defines it as a bool in its schema
15:10:08 mriedem we don't use glanceclient
15:10:08 sean-k-mooney http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/v2/image_schema.py#n216
15:10:15 sean-k-mooney oh ok
15:10:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
15:10:18 mriedem i think we only use ksa now
15:10:48 sean-k-mooney we still import glancclient
15:11:27 mriedem oh yeah i guess we do,
15:11:35 mriedem we use ksa to get the session adapter thing to construct glanceclient
15:12:04 mriedem anyway, it'd be cool if apis didn't return things you didn't ask for...
15:12:37 diliprenkila mriedem: yes
15:12:41 sean-k-mooney so could we modify glance client to munge the values form strings to bools so we did not have to handel this field ourselves
15:15:30 mriedem i'm sure if we complained to the glance team about this, they'd say "you should be using the schema provided with the image"
15:15:35 mriedem which we aren't
15:15:55 mriedem there is only one image field we deal with the schema and that's disk_format
15:16:20 mriedem https://review.openstack.org/#/c/375875/
15:19:53 diliprenkila mriedem: so we should fix the os_hidden type in nova ? not in glance
15:20:56 openstackgerrit Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195
15:21:59 mriedem diliprenkila: i don't think there is probably anything to change in glance,
15:22:14 mriedem but i'd like to know why tempest isn't failing with this, but i need to know the recreate steps,
15:22:15 openstackgerrit Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972
15:22:30 mriedem because tempest has very basic tests where it creates a server and then creates a snapshot of that server,
15:22:40 mriedem which i would think should cause this failure
15:26:27 mriedem maybe tempest isn't using image api v2.7/
15:26:29 mriedem ?
15:27:59 diliprenkila mridem: may be
15:28:06 artom mriedem, I think by default it goes to the lowest microversion
15:28:21 mriedem glance doesn't have microversions but...
15:28:22 artom Unless the test specifies min_microversion (or just microversion?)
15:28:33 artom Oh, so just endpoints?
15:28:56 mriedem i need a mordred
15:29:14 mordred I didn't do it
15:29:19 mriedem image api versions, go!
15:29:23 mriedem as in, wtf
15:29:25 artom You're holding out for a mordred 'till the end of the night
15:29:30 mriedem does the user opt into those or you just get what the server has available?
15:29:31 diliprenkila mriedem: i am using nova: 18.0.0 , glance: 2.9.1
15:29:54 mordred they're silly. there is no selection - you just get the API described by the highest number in that list
15:30:06 mordred so - basically - ignore the thing after the 2.
15:30:17 mriedem so if glance is rocky, i get 2.7
15:30:22 mriedem https://developer.openstack.org/api-ref/image/versions/index.html#version-history
15:30:26 mordred yeah
15:30:40 mriedem then i don't know why tempest would not fail on https://bugs.launchpad.net/nova/+bug/1806239
15:30:40 openstack Launchpad bug 1806239 in OpenStack Compute (nova) "nova-api should handle type conversion while creating server snapshots " [Undecided,New]
15:31:07 mordred nova is still using glanceclient right?
15:31:12 mriedem yeah
15:31:33 mordred yeah - tempest uses direct rest calls - so it's possible glanceclient is doing something wrong. or tempest is doing something wrong
15:32:29 mriedem well, tempest would just create a server and tell nova to snapshot it
15:32:33 mriedem and then nova will use glanceclient
15:32:52 mriedem if that's all it takes to tickle this with rocky glance, i'm not sure why tempest wouldn't blow up
15:33:37 mriedem anyway, i'll wait for diliprenkila to provide recreate steps
15:33:59 mordred oh. gotcha
15:34:05 mordred yeah. that's super weird
15:35:10 ShilpaSD Hi All, facing issue while stack, E: Sub-process /usr/bin/dpkg returned an error code (1), any suggestions to resolve this?
15:37:28 dansmith mriedem: so on that rpc logging thing at startup, do we think the actual query is slowing things down, or the logging of the pointless message?
15:37:47 mriedem idk
15:38:26 mriedem looking at logs
15:39:20 mriedem well we take 2 seconds dumping our gd config options :)
15:39:35 dansmith which should be mostly just log traffic, right?
15:39:40 mriedem yeah
15:39:41 dansmith so maybe the logging is the thing
15:39:42 dansmith also
15:39:44 mriedem we start loading extensions at Dec 05 20:14:00.919520
15:40:03 dansmith you know that if we were to actually start compute first, we would cache the service version the way we want and avoid the multiple lookups
15:40:24 mriedem looks like we're done loading extensions at Dec 05 20:14:27.718587
15:40:43 mriedem so there is another thing here,
15:40:49 mriedem the rpcapi client does the version query thing,
15:41:04 mriedem but API also constructs a SchedulerReportClient per instance, which apparently uses a lock
15:41:04 mriedem Dec 05 20:14:27.687766 ubuntu-xenial-ovh-bhs1-0000959981 devstack@n-api.service[23459]: DEBUG oslo_concurrency.lockutils [None req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock "placement_client" acquired by "nova.scheduler.client.report._create_client" :: waited 0.000s {{(pid=23462) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:327}}
15:41:49 mriedem so...
15:41:55 dansmith I also just noticed/remembered that the multi-cell version of this does not cache at all
15:41:57 mriedem and that's an in-memory lock
15:44:28 mriedem heh remember how efried_cya_jan removed that lazy-load scheduler report client stuff? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L256
15:44:45 dansmith yeah
15:44:51 efried_cya_jan uh oh
15:45:17 dansmith mriedem: is there a warn-once pattern I should be using for logs?
15:46:23 mriedem dansmith: i think just a global
15:46:31 dansmith ack
15:46:48 mriedem i'll strip out a separate bug for this report client init thing
15:51:01 mriedem https://bugs.launchpad.net/nova/+bug/1807219
15:51:01 openstack Launchpad bug 1807219 in OpenStack Compute (nova) "SchedulerReporClient init slows down nova-api startup" [Medium,Triaged]
15:52:22 cdent mdbooth: is that ^ related at all to the slow down you were seeing in your explorations yesterday(?)?
15:53:20 efried_cya_jan mriedem: We ought to singleton that guy. If we're not having caching conflicts, it can only be out of luck because the API obj isn't doing anything that touches the cache.
15:53:47 edleafe efried_cya_jan: more like efried_big_fat_liar
15:54:45 efried_cya_jan I'm here for like another 20 minutes today. This does two things: 1) keeps my inbox down to triple digits; 2) makes you scared to talk about me behind my back.
15:55:03 mriedem i intentionally talked about you to summon you
15:55:06 mdbooth cdent: Only if we also do this during server create. Possible. The number of API objects created was insane.
15:55:08 kaisers_ stephenfin: mdbooth: Updated https://review.openstack.org/#/c/554195/ as discussed yesterday
15:55:15 edleafe Oh, it's more fun to talk about you to your face!
15:56:23 efried_cya_jan mriedem: Is the removal of lazy-load causing (or even related to) that one-reportclient-per-API?
15:57:59 mriedem efried_cya_jan: not sure, we might have always been loading it during API init
15:58:41 mriedem but i think it would have done the lazy-load

Earlier   Later